[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 736277705 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003365] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010146] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013085] *** VALIDATE selinux *** [ 0.022780] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027757] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030130] *** VALIDATE tmpfs *** [ 0.032377] *** VALIDATE proc *** [ 0.034005] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.037044] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044076] debug: unmapping init [mem 0xffffffff8d459000-0xffffffff8d460fff] [ 0.047257] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048798] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053019] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056382] rcu: Hierarchical SRCU implementation. [ 0.058542] smp: Bringing up secondary CPUs ... [ 0.059609] x86: Booting SMP configuration: [ 0.060026] .... node #0, CPUs: #1 #2 #3 [ 0.071654] smp: Brought up 1 node, 4 CPUs [ 0.073032] smpboot: Max logical packages: 1 [ 0.074014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105674] node 0 deferred pages initialised in 29ms [ 0.108267] devtmpfs: initialized [ 0.109152] x86/mm: Memory block size: 128MB [ 0.111198] gcov: version magic: 0x41383552 [ 0.113326] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.115068] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.118280] pinctrl core: initialized pinctrl subsystem [ 0.119204] [ 0.119639] ************************************************************* [ 0.121011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.123010] ** ** [ 0.125009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.127010] ** ** [ 0.129009] ** This means that this kernel is built to expose internal ** [ 0.131011] ** IOMMU data structures, which may compromise security on ** [ 0.133013] ** your system. ** [ 0.136012] ** ** [ 0.139014] ** If you see this message and you are not debugging the ** [ 0.142022] ** kernel, report this immediately to your vendor! ** [ 0.145014] ** ** [ 0.146012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148011] ************************************************************* [ 0.151796] NET: Registered protocol family 16 [ 0.153519] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.157064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.161069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165166] cpuidle: using governor menu [ 0.166939] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170571] PCI: Using configuration type 1 for base access [ 0.173220] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181088] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184064] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188033] cryptd: max_cpu_qlen set to 1000 [ 0.207062] ACPI: Added _OSI(Module Device) [ 0.212018] ACPI: Added _OSI(Processor Device) [ 0.214017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221695] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229215] ACPI: Interpreter enabled [ 0.231059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233012] ACPI: Using IOAPIC for interrupt routing [ 0.235137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.241418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254088] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260076] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270180] acpiphp: Slot [2] registered [ 0.271174] acpiphp: Slot [5] registered [ 0.272091] acpiphp: Slot [6] registered [ 0.273114] acpiphp: Slot [3] registered [ 0.275241] acpiphp: Slot [4] registered [ 0.276077] acpiphp: Slot [7] registered [ 0.277070] acpiphp: Slot [8] registered [ 0.279095] acpiphp: Slot [9] registered [ 0.291110] acpiphp: Slot [10] registered [ 0.293147] acpiphp: Slot [11] registered [ 0.294058] acpiphp: Slot [12] registered [ 0.295059] acpiphp: Slot [13] registered [ 0.296096] acpiphp: Slot [14] registered [ 0.297106] acpiphp: Slot [15] registered [ 0.298151] acpiphp: Slot [16] registered [ 0.299089] acpiphp: Slot [17] registered [ 0.301124] acpiphp: Slot [18] registered [ 0.302104] acpiphp: Slot [19] registered [ 0.304112] acpiphp: Slot [20] registered [ 0.305067] acpiphp: Slot [21] registered [ 0.306067] acpiphp: Slot [22] registered [ 0.306959] acpiphp: Slot [23] registered [ 0.308082] acpiphp: Slot [24] registered [ 0.309020] acpiphp: Slot [25] registered [ 0.309814] acpiphp: Slot [26] registered [ 0.311083] acpiphp: Slot [27] registered [ 0.311860] acpiphp: Slot [28] registered [ 0.313116] acpiphp: Slot [29] registered [ 0.314089] acpiphp: Slot [30] registered [ 0.315207] acpiphp: Slot [31] registered [ 0.317060] PCI host bridge to bus 0000:00 [ 0.318014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.320020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.324030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.326000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.326458] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.334016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.338054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.341021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.343019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.346019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.353085] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356125] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.357029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.361298] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.366014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.378953] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.384013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.389574] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.395013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.402020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.439026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.466047] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.488022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.523252] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.524412] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.526488] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.529389] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.531213] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.534189] iommu: Default domain type: Passthrough [ 0.536413] SCSI subsystem initialized [ 0.538174] ACPI: bus type USB registered [ 0.539136] usbcore: registered new interface driver usbfs [ 0.541244] usbcore: registered new interface driver hub [ 0.544134] usbcore: registered new device driver usb [ 0.545173] pps_core: LinuxPPS API ver. 1 registered [ 0.547014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.550090] PTP clock support registered [ 0.552214] EDAC MC: Ver: 3.0.0 [ 0.555152] PCI: Using ACPI for IRQ routing [ 0.557563] NetLabel: Initializing [ 0.559022] NetLabel: domain hash size = 128 [ 0.561017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.563137] NetLabel: unlabeled traffic allowed by default [ 0.566508] vgaarb: loaded [ 0.568497] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.571023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.578626] clocksource: Switched to clocksource kvm-clock [ 0.708952] VFS: Disk quotas dquot_6.6.0 [ 0.710559] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.713475] *** VALIDATE ramfs *** [ 0.714568] *** VALIDATE hugetlbfs *** [ 0.716229] pnp: PnP ACPI init [ 0.718675] pnp: PnP ACPI: found 6 devices [ 0.734449] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.738502] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.741597] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.744551] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.748125] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.752055] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.756309] NET: Registered protocol family 2 [ 0.758983] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.765680] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.769921] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.793476] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.800495] TCP: Hash tables configured (established 65536 bind 65536) [ 0.804387] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.811724] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.815921] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.819696] NET: Registered protocol family 1 [ 0.823047] RPC: Registered named UNIX socket transport module. [ 0.826074] RPC: Registered udp transport module. [ 0.829215] RPC: Registered tcp transport module. [ 0.830884] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.833558] NET: Registered protocol family 44 [ 0.835095] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.837585] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.839497] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.841677] PCI: CLS 0 bytes, default 64 [ 0.843379] Unpacking initramfs... [ 2.430800] debug: unmapping init [mem 0xffff96507cc64000-0xffff96507ffcffff] [ 2.437250] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.439063] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.443128] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.962881] Initialise system trusted keyrings [ 2.964638] Key type blacklist registered [ 2.966455] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.987454] zbud: loaded [ 2.991511] *** VALIDATE nfs *** [ 2.992151] *** VALIDATE nfs4 *** [ 2.993534] pstore: using deflate compression [ 2.996127] Platform Keyring initialized [ 3.096059] NET: Registered protocol family 38 [ 3.097423] Key type asymmetric registered [ 3.098273] Asymmetric key parser 'x509' registered [ 3.099367] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.101341] io scheduler mq-deadline registered [ 3.102927] io scheduler kyber registered [ 3.104971] io scheduler bfq registered [ 3.107198] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.109568] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.112465] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.115386] ACPI: Power Button [PWRF] [ 3.120504] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.126249] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.137066] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.165178] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.201749] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.242522] Non-volatile memory driver v1.3 [ 3.244636] Linux agpgart interface v0.103 [ 3.281764] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.286512] vda: detected capacity change from 0 to 75100160 [ 3.303663] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.306988] vdb: detected capacity change from 0 to 1073741824 [ 3.314226] libphy: Fixed MDIO Bus: probed [ 3.336989] usbcore: registered new interface driver usbserial_generic [ 3.340495] usbserial: USB Serial support registered for generic [ 3.343409] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.348881] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.350505] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.352714] mousedev: PS/2 mouse device common for all mice [ 3.356500] rtc_cmos 00:05: RTC can wake from S4 [ 3.359540] rtc_cmos 00:05: registered as rtc0 [ 3.361813] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.362293] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.370275] intel_pstate: CPU model not supported [ 3.375518] hid: raw HID events driver (C) Jiri Kosina [ 3.376618] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.378080] usbcore: registered new interface driver usbhid [ 3.378086] usbhid: USB HID core driver [ 3.378392] drop_monitor: Initializing network drop monitor service [ 3.384054] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.385308] Initializing XFRM netlink socket [ 3.392762] NET: Registered protocol family 10 [ 3.397396] Segment Routing with IPv6 [ 3.398309] NET: Registered protocol family 17 [ 3.400901] mpls_gso: MPLS GSO support [ 3.409164] RAS: Correctable Errors collector initialized. [ 3.410612] AVX version of gcm_enc/dec engaged. [ 3.411618] AES CTR mode by8 optimization enabled [ 3.483536] sched_clock: Marking stable (3483508030, 0)->(4435260974, -951752944) [ 3.487677] registered taskstats version 1 [ 3.490357] Loading compiled-in X.509 certificates [ 3.492534] zswap: loaded using pool lzo/zbud [ 3.521415] Key type big_key registered [ 3.534501] Key type encrypted registered [ 3.536191] ima: No TPM chip found, activating TPM-bypass! [ 3.537744] ima: Allocated hash algorithm: sha1 [ 3.539610] ima: No architecture policies found [ 3.541341] evm: Initialising EVM extended attributes: [ 3.543359] evm: security.selinux [ 3.543961] evm: security.ima [ 3.544700] evm: security.capability [ 3.545553] evm: HMAC attrs: 0x1 [ 3.547137] rtc_cmos 00:05: setting system clock to 2026-09-13 02:04:11 UTC (1789265051) [ 3.551634] debug: unmapping init [mem 0xffffffff8e403000-0xffffffff8e5fffff] [ 3.554437] debug: unmapping init [mem 0xffffffff8d182000-0xffffffff8d458fff] [ 3.564118] Write protecting the kernel read-only data: 28672k [ 3.566906] debug: unmapping init [mem 0xffffffff8b803000-0xffffffff8b9fffff] [ 3.568942] debug: unmapping init [mem 0xffffffff8c114000-0xffffffff8c1fffff] [ 3.601609] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.612606] systemd[1]: Detected virtualization kvm. [ 3.615496] systemd[1]: Detected architecture x86-64. [ 3.618325] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.645244] systemd[1]: No hostname configured. [ 3.650328] systemd[1]: Set hostname to . [ 3.652059] random: systemd: uninitialized urandom read (16 bytes read) [ 3.654523] systemd[1]: Initializing machine ID from random generator. [ 3.767883] random: ln: uninitialized urandom read (6 bytes read) [ 3.899185] random: systemd: uninitialized urandom read (16 bytes read) [ 3.900578] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.905328] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.908564] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.868202] device-mapper: uevent: version 1.0.3 [ 4.870517] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted Kernel Configuration File System. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.591532] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.861440] scsi host0: ata_piix [ 5.885226] scsi host1: ata_piix [ 5.889120] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.894963] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.115234] random: fast init done [ 10.052474] random: crng init done [ 10.053987] random: 7 urandom warning(s) missed due to ratelimiting [ 10.635640] dracut-initqueue[569]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.206783] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.655857] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.999073] SELinux: Disabled at runtime. [ 14.065432] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.075365] systemd[1]: Detected virtualization kvm. [ 14.077312] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.758206] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.761778] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.767975] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.773427] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.776676] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.786345] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.793287] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ 14.943314] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 15.719561] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.145126] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.281496] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.467402] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.539236] EDAC sbridge: Ver: 1.1.2 [ 18.137531] Key type dns_resolver registered [ 18.481405] NFS: Registering the id_resolver key type [ 18.483833] Key type id_resolver registered [ 18.485846] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg133-client login: [ 77.823812] libcfs: loading out-of-tree module taints kernel. [ 78.003028] Key type ._llcrypt registered [ 78.005548] Key type .llcrypt registered [ 78.916361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 78.941621] alg: No test for adler32 (adler32-zlib) [ 80.669485] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 81.583900] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 83.339737] Key type lgssc registered [ 85.482655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 209.382504] Lustre: Mounted lustre-client - version 2.17.57_106_gd81f50d [ 214.223124] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.694396] hrtimer: interrupt took 4387513 ns [ 229.090991] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing check_logdir /tmp/testlogs/ [ 234.002265] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing yml_node [ 234.978674] Lustre: lustre-OST0000-osc-ffff9650d0315800: disconnect after 23s idle [ 238.709909] Lustre: DEBUG MARKER: Client: 2.17.57.106 [ 241.096128] Lustre: DEBUG MARKER: MDS: 2.17.57.106 [ 243.524871] Lustre: DEBUG MARKER: OSS: 2.17.57.106 [ 244.878299] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 12 22:08:11 EDT 2026 [ 259.105194] Lustre: Unmounted lustre-client [ 282.790887] Key type lgssc unregistered [ 283.016499] LNet: 8410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.020497] LNetError: 8410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 283.049188] LNet: Removed LNI 192.168.201.33@tcp [ 283.618335] Key type .llcrypt unregistered [ 283.622174] Key type ._llcrypt unregistered [ 306.951505] Key type ._llcrypt registered [ 306.953592] Key type .llcrypt registered [ 307.124156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 307.139068] alg: No test for adler32 (adler32-zlib) [ 308.096755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 308.179867] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 308.207509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 312.689043] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 312.915048] LNet: 9282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.925834] LNetError: 9282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.980878] LNet: Removed LNI 192.168.201.33@tcp [ 313.480032] Key type .llcrypt unregistered [ 313.481384] Key type ._llcrypt unregistered [ 319.248611] Key type ._llcrypt registered [ 319.249965] Key type .llcrypt registered [ 320.322738] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 22:09:27 (1789265367) [ 320.598757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 320.613302] alg: No test for adler32 (adler32-zlib) [ 321.595350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.728463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.748871] LNet: 9698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.771953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.814921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 321.835263] LNet: 9705:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 329.148920] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 22:09:35 (1789265375) [ 329.534818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.563494] LNet: 10282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.587868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.679967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 336.876487] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 22:09:43 (1789265383) [ 337.122469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.144777] LNet: 10869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.169934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.254271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 344.469794] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 22:09:50 (1789265390) [ 344.711355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.759480] LNet: 11454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.819225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.999163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 353.346900] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 22:09:59 (1789265399) [ 353.477733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.523568] LNet: 12035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.586604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.736829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 354.457940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 362.931628] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 22:10:09 (1789265409) [ 363.124225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.190293] LNet: 12627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.249611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.418226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.797779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 364.832280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 372.697070] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 22:10:19 (1789265419) [ 372.768932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.824477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.918635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 379.836244] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 22:10:25 (1789265425) [ 380.209538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.239181] LNet: 13812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.249854] LNet: 13812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 380.273312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.367545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 380.403838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 381.075596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 381.147149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 381.570196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 381.588520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 381.992571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 382.052270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 382.360476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 382.383852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 382.697626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 382.717859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 382.993346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 383.026022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 383.250717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 383.273525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 383.500501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 383.520811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 383.810159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 383.870633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 384.223634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 384.245934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 384.960303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 385.105384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 385.720169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 385.770571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 394.684786] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 22:10:41 (1789265441) [ 394.935468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.985481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.106755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 395.157658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 402.026698] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 22:10:48 (1789265448) [ 402.229116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.301142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.389591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 402.435785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 409.078877] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 22:10:55 (1789265455) [ 409.231415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.350250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.437920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 409.483100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 415.813465] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 22:11:02 (1789265462) [ 416.093896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.112328] LNet: 16255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.118841] LNet: 16255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 416.154447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.283371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 416.351631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 423.417483] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 22:11:09 (1789265469) [ 423.625768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.710109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.829281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 423.856102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 430.112565] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 22:11:16 (1789265476) [ 430.274698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.330731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.424870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 430.473494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 436.972508] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 22:11:23 (1789265483) [ 437.204283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.261889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.381666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 437.773158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 444.414307] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 22:11:31 (1789265491) [ 444.521878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.560713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.767397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.075732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.244808] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 22:11:38 (1789265498) [ 452.456166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.562674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.730118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.944963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 459.235826] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 22:11:46 (1789265506) [ 459.306740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.363858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.713598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 459.897441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.702875] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 22:11:52 (1789265512) [ 465.965927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.020560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.101227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.150988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.302659] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 22:11:58 (1789265518) [ 472.392735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.452527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.582656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 472.640889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 480.202837] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 22:12:06 (1789265526) [ 480.300109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.329357] LNet: 21690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 480.347668] LNet: 21690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 480.388448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.530309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 486.393961] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 22:12:13 (1789265533) [ 486.682749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.724491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.806716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 494.738284] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 22:12:20 (1789265540) [ 494.903381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.930891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.992766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 495.311226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 502.297788] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 22:12:28 (1789265548) [ 502.448868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.525976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.731217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 502.785679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 512.213970] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 22:12:38 (1789265558) [ 512.383485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.477654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.620779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 512.684356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 519.184569] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 22:12:45 (1789265565) [ 519.357136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.440994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.488359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 519.511300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 525.524544] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 22:12:52 (1789265572) [ 525.728390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.777296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.842418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 525.873589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 532.469327] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 22:12:59 (1789265579) [ 532.713179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.773469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.835262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 532.910479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 532.925833] LNetError: 25813:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 532.962596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 532.988063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 539.409531] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 22:13:05 (1789265585) [ 539.747808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.786442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.869527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LUh5ZcrSVm [ 539.938598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LUh5ZcrSVm [ 539.966389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LUh5ZcrSVm [ 546.677120] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 22:13:13 (1789265593) [ 546.771573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.822451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.908567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 546.964035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 547.001095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 554.634590] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 22:13:20 (1789265600) [ 554.918897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.974398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.074738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 555.102877] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 556.858744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 558.627359] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 560.776733] Key type lgssc registered [ 562.501347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 567.091692] LNetError: 27574:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.33@tcp99 has no available nets [ 570.207143] Key type lgssc unregistered [ 570.675124] LNetError: 28680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 570.697957] LNet: Removed LNI 192.168.201.33@tcp [ 571.497084] Key type .llcrypt unregistered [ 571.500084] Key type ._llcrypt unregistered [ 579.142867] Key type ._llcrypt registered [ 579.147789] Key type .llcrypt registered [ 580.743930] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 22:13:46 (1789265626) [ 581.291370] Key type .llcrypt unregistered [ 581.293498] Key type ._llcrypt unregistered [ 589.306098] Key type ._llcrypt registered [ 589.308205] Key type .llcrypt registered [ 589.499846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 589.552945] alg: No test for adler32 (adler32-zlib) [ 590.523613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.606372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 590.651703] LNet: 29591:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 590.665429] LNet: 29591:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 590.672868] LNet: 29591:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 590.700438] LNet: Message forwarding will be enabled in 2s [ 593.760235] LNet: Message forwarding enabled [ 594.316802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 594.345465] LNet: Message forwarding will stop in 2s to 6s [ 597.867559] LNet: Message forwarding disabled [ 598.924451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 598.961514] LNet: 29690:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 598.974934] LNet: 29690:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 598.980342] LNet: 29690:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 599.015141] LNet: Message forwarding will be enabled in 2s [ 599.292349] LNet: 29703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.098903] Key type .llcrypt unregistered [ 600.100968] Key type ._llcrypt unregistered [ 613.481431] Key type ._llcrypt registered [ 613.483914] Key type .llcrypt registered [ 614.748343] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 22:14:21 (1789265661) [ 615.216958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 615.230723] alg: No test for adler32 (adler32-zlib) [ 616.183816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.292638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 623.178618] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 22:14:29 (1789265669) [ 623.293644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.327322] LNet: 31110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.378921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.508164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 623.546282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 623.584132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 623.617827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 623.673375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 623.727804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 623.800848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 623.851076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 623.914842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 623.981289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 624.043818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 624.083391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 624.125187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 624.173077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 624.245656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 624.309237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 624.375752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 624.427360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 624.460842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 624.482911] LNetError: 31160:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 631.710954] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 22:14:38 (1789265678) [ 631.811957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.852540] LNet: 31723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.883910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.009592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.749533] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 22:14:45 (1789265685) [ 640.747303] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg133-server [ 642.738047] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 22:14:49 (1789265689) [ 642.958329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.026765] LNet: 32662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.056209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.148979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 643.182539] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 644.151343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[34-38]@tcp [ 644.197927] LNet: 32796:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 644.275151] LNetError: 32666:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.36@tcp is being used as a gateway but routing feature is not turned on [ 644.315597] LNetError: 32666:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.36@tcp has gone from up to down [ 651.768352] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 22:14:58 (1789265698) [ 653.373227] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg133-server [ 655.348597] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 22:15:01 (1789265701) [ 655.458097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.503572] LNet: 33725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.550995] LNetError: 33725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.588057] LNet: Removed LNI 192.168.201.33@tcp [ 655.648192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.718731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 655.743750] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 656.577500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[34-39/2]@tcp [ 656.615183] LNet: 33850:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 656.626145] LNet: 33850:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 656.655170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[34-39/2]@tcp [ 656.674290] LNetError: 33736:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.38@tcp is being used as a gateway but routing feature is not turned on [ 656.683699] LNetError: 33736:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 656.686592] LNetError: 33736:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.38@tcp has gone from up to down [ 656.692677] LNetError: 33736:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 663.535979] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 22:15:09 (1789265709) [ 663.651971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.688247] LNet: 34434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.714190] LNetError: 34434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.772633] LNet: Removed LNI 192.168.201.33@tcp [ 663.809247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.979904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 664.032961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.081438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.145441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.203081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.266618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.341588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 664.442317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.518704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.659187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.806930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.893893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.978566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 665.047733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.065430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.118531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.198306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.234605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.270842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 665.330404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.380069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.425526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.524841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.552026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.627306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 665.708133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.726220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.758555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.821148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.852546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.888957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 665.930273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.964541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.019610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.860339] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 22:15:20 (1789265720) [ 676.279090] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg133-server [ 678.174351] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 22:15:24 (1789265724) [ 681.040516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg133-server [ 682.847761] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 22:15:29 (1789265729) [ 683.002664] LNet: 35836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.009942] LNet: 35836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 684.131759] Key type .llcrypt unregistered [ 684.140651] Key type ._llcrypt unregistered [ 693.905377] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 694.259141] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 694.264743] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 694.268873] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 695.777769] Key type ._llcrypt registered [ 695.781600] Key type .llcrypt registered [ 695.969418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.993120] alg: No test for adler32 (adler32-zlib) [ 696.976697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.117236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.166966] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 698.288837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 698.338743] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 701.314935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 701.331363] LNet: Removed LNI 10.1.2.3@tcp [ 701.532817] LNet: 36552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.543360] LNetError: 36552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.568887] LNet: Removed LNI 192.168.201.33@tcp [ 702.648268] Key type .llcrypt unregistered [ 702.652690] Key type ._llcrypt unregistered [ 721.082351] Key type ._llcrypt registered [ 721.084243] Key type .llcrypt registered [ 722.737916] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 22:16:08 (1789265768) [ 723.472973] Key type .llcrypt unregistered [ 723.475616] Key type ._llcrypt unregistered [ 731.671656] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 731.817985] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 731.826031] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 731.832138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 734.435082] Key type ._llcrypt registered [ 734.440703] Key type .llcrypt registered [ 734.587763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 734.600274] alg: No test for adler32 (adler32-zlib) [ 735.537590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.589946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.605026] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 735.980338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 735.994468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 738.606790] LNet: 38043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.610906] LNetError: 38043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 738.616466] LNet: Removed LNI 10.1.2.3@tcp [ 738.634819] LNet: Removed LNI 192.168.201.33@tcp [ 739.315991] Key type .llcrypt unregistered [ 739.319549] Key type ._llcrypt unregistered [ 752.634812] Key type ._llcrypt registered [ 752.637409] Key type .llcrypt registered [ 754.381151] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 22:16:40 (1789265800) [ 755.150358] Key type .llcrypt unregistered [ 755.152111] Key type ._llcrypt unregistered [ 763.793280] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 764.165809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 764.189818] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 764.204240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 767.065835] Key type ._llcrypt registered [ 767.067533] Key type .llcrypt registered [ 767.306390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 767.323701] alg: No test for adler32 (adler32-zlib) [ 768.326412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.424106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 768.468047] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 770.575260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 770.591709] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 773.162267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 773.208521] LNet: Removed LNI 10.1.2.3@tcp [ 773.241711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 773.287776] LNet: Removed LNI 10.1.2.31@tcp [ 773.304341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 773.350872] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 774.332449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 774.378720] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 775.633444] LNet: 39959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.642559] LNetError: 39959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 775.657241] LNet: Removed LNI 10.1.2.31@tcp [ 775.679224] LNet: Removed LNI 10.1.2.3@tcp [ 776.623529] Key type .llcrypt unregistered [ 776.626184] Key type ._llcrypt unregistered [ 791.509413] Key type ._llcrypt registered [ 791.511349] Key type .llcrypt registered [ 793.048957] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 22:17:19 (1789265839) [ 793.718427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 793.756733] alg: No test for adler32 (adler32-zlib) [ 794.789388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.852439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 794.874465] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 795.774703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.799826] LNet: 40909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.804853] LNetError: 40909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.819184] LNet: Removed LNI 192.168.201.33@tcp [ 795.852867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.937134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.33@tcp [ 795.993261] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 796.859112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.33@tcp [ 796.905253] LNet: Removed LNI 192.168.201.33@tcp [ 796.969807] LNet: 41044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.690768] Key type .llcrypt unregistered [ 797.693693] Key type ._llcrypt unregistered [ 811.241653] Key type ._llcrypt registered [ 811.243084] Key type .llcrypt registered [ 812.721818] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 22:17:38 (1789265858) [ 815.096603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg133-server [ 817.411354] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 22:17:43 (1789265863) [ 818.293529] Key type .llcrypt unregistered [ 818.295647] Key type ._llcrypt unregistered [ 825.860397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 826.146436] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 826.154607] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 826.171595] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 828.939436] Key type ._llcrypt registered [ 828.941076] Key type .llcrypt registered [ 829.187571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 829.222034] alg: No test for adler32 (adler32-zlib) [ 830.199418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.331317] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 831.493550] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 833.970043] LNet: 42877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.984623] LNetError: 42877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.996440] LNet: Removed LNI 192.168.201.33@tcp1 [ 834.019289] LNet: Removed LNI 10.1.2.3@tcp2 [ 834.627409] Key type .llcrypt unregistered [ 834.630126] Key type ._llcrypt unregistered [ 847.577874] Key type ._llcrypt registered [ 847.581501] Key type .llcrypt registered [ 848.852618] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 22:18:15 (1789265895) [ 849.672981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 849.717995] alg: No test for adler32 (adler32-zlib) [ 850.677364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.733273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 850.759137] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 852.101517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.123859] LNet: 43844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.133719] LNetError: 43844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.149402] LNet: Removed LNI 192.168.201.33@tcp [ 852.177443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.235621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 852.257170] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 859.589932] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 22:18:26 (1789265906) [ 859.878995] LNet: 44436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.900231] LNetError: 44436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.915415] LNet: Removed LNI 192.168.201.33@tcp [ 860.562559] Key type .llcrypt unregistered [ 860.564885] Key type ._llcrypt unregistered [ 867.692480] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 867.933672] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 867.939625] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 867.943677] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 870.579848] Key type ._llcrypt registered [ 870.582588] Key type .llcrypt registered [ 870.883533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.909425] alg: No test for adler32 (adler32-zlib) [ 872.040960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.113426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.152136] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 872.912650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 872.935501] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 875.850960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.884170] LNet: 45174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.898441] LNetError: 45174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 875.906390] LNet: Removed LNI 10.1.2.3@tcp [ 875.927536] LNet: Removed LNI 192.168.201.33@tcp [ 875.950315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.996554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 876.022595] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 876.026222] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 876.606955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.623498] LNet: 45201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.631307] LNetError: 45201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 876.638426] LNetError: 45201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 876.644676] LNet: Removed LNI 10.1.2.3@tcp [ 876.663355] LNet: Removed LNI 192.168.201.33@tcp [ 876.689445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.755593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 876.784743] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 876.789768] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 877.471762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 877.488639] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 878.572372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.592613] LNet: 45363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.611116] LNetError: 45363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 878.623498] LNetError: 45363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 878.633044] LNet: Removed LNI 10.1.2.3@tcp [ 878.634545] LNet: Removed LNI 192.168.201.33@tcp [ 878.644363] LNet: Removed LNI 192.168.201.33@tcp2 [ 878.673958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.763896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 878.827956] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 878.838370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 878.849743] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 886.948458] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 22:18:53 (1789265933) [ 887.131208] LNet: 45959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.146603] LNetError: 45959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 887.162497] LNetError: 45959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 887.180664] LNet: Removed LNI 10.1.2.3@tcp [ 887.183156] LNet: Removed LNI 192.168.201.33@tcp [ 887.199278] LNet: Removed LNI 192.168.201.33@tcp2 [ 887.843388] Key type .llcrypt unregistered [ 887.845466] Key type ._llcrypt unregistered [ 894.439422] Key type ._llcrypt registered [ 894.440904] Key type .llcrypt registered [ 894.627254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.641604] alg: No test for adler32 (adler32-zlib) [ 895.615980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.708861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.732428] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 896.526797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.550909] LNet: 46353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.555537] LNetError: 46353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.571854] LNet: Removed LNI 192.168.201.33@tcp [ 896.594258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.659063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 896.683162] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 897.252437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.285676] LNet: 46383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.291716] LNetError: 46383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.317815] LNet: Removed LNI 192.168.201.33@tcp [ 897.344807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.447266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 897.476876] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 897.918335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.960837] LNet: Removed LNI 192.168.201.33@tcp [ 897.984687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.054643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 898.085057] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 898.690708] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 898.842234] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 898.855180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 898.869440] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 901.259524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.282382] LNet: 46493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.289850] LNet: 46493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 901.299747] LNetError: 46493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.313813] LNetError: 46493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 901.343463] LNet: Removed LNI 192.168.201.33@tcp [ 901.379766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.477590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 901.513543] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 908.455913] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 22:19:14 (1789265954) [ 908.567803] LNet: 47086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.579318] LNetError: 47086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.596147] LNet: Removed LNI 192.168.201.33@tcp [ 909.342905] Key type .llcrypt unregistered [ 909.344681] Key type ._llcrypt unregistered [ 916.610768] Key type ._llcrypt registered [ 916.614447] Key type .llcrypt registered [ 916.872909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 916.888893] alg: No test for adler32 (adler32-zlib) [ 917.840740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.910788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 917.935755] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 918.785843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.798352] LNet: 47488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.808047] LNetError: 47488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.817819] LNet: Removed LNI 192.168.201.33@tcp [ 918.831751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.879304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 918.894911] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 925.305736] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 22:19:31 (1789265971) [ 925.445250] LNet: 48072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.450212] LNetError: 48072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.462883] LNet: Removed LNI 192.168.201.33@tcp [ 926.294500] Key type .llcrypt unregistered [ 926.298348] Key type ._llcrypt unregistered [ 932.999953] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 933.195082] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 933.200336] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 933.208682] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 935.869293] Key type ._llcrypt registered [ 935.871492] Key type .llcrypt registered [ 936.005823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 936.016221] alg: No test for adler32 (adler32-zlib) [ 936.964895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.044817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 937.079598] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 937.921755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.936082] LNet: 48528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.940774] LNetError: 48528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.952882] LNet: Removed LNI 192.168.201.33@tcp [ 937.967797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.018392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 938.037450] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 944.546775] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 22:19:51 (1789265991) [ 944.673200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.723515] LNet: 49117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.735177] LNetError: 49117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.765034] LNet: Removed LNI 192.168.201.33@tcp [ 944.781884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.927901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 951.729284] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 22:19:57 (1789265997) [ 951.846907] LNet: 49700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.775814] Key type .llcrypt unregistered [ 952.779453] Key type ._llcrypt unregistered [ 959.926345] Key type ._llcrypt registered [ 959.929083] Key type .llcrypt registered [ 960.125787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.142923] alg: No test for adler32 (adler32-zlib) [ 961.119773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.189833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.243795] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 962.226382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.239758] LNet: 50103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.242785] LNetError: 50103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.251573] LNet: Removed LNI 192.168.201.33@tcp [ 962.267230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.322130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 962.348787] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 970.558363] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 22:20:16 (1789266016) [ 970.624567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.642935] LNet: 50688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.651892] LNetError: 50688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.665169] LNet: Removed LNI 192.168.201.33@tcp [ 970.690117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.834670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 976.894221] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 22:20:23 (1789266023) [ 978.618665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 980.374146] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 22:20:26 (1789266026) [ 982.046171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 984.062506] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 22:20:30 (1789266030) [ 985.883846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 987.834955] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 22:20:34 (1789266034) [ 989.733527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 991.292606] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 22:20:37 (1789266037) [ 992.712230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 994.348407] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 22:20:41 (1789266041) [ 994.502226] LNet: 52744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.046707] Key type .llcrypt unregistered [ 995.049220] Key type ._llcrypt unregistered [ 1001.342511] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1001.469350] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1001.479634] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1001.483457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1004.142902] Key type ._llcrypt registered [ 1004.145341] Key type .llcrypt registered [ 1004.399256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1004.420348] alg: No test for adler32 (adler32-zlib) [ 1005.465330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.597218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1005.619704] LNet: Added LNI 192.168.201.33@tcp [8/512/0/180] [ 1005.625441] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1005.629299] LNet: 53066:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1005.715783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.734356] LNet: 53085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.737920] LNetError: 53085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.742904] LNet: Removed LNI 10.1.2.3@tcp [ 1005.745325] LNetError: 53085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.760953] LNet: Removed LNI 192.168.201.33@tcp [ 1005.782692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.850480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1005.880295] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1005.963994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.977851] LNet: 53109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.982653] LNetError: 53109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.996333] LNet: Removed LNI 192.168.201.33@tcp [ 1006.042296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.126099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.166632] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1006.178507] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1006.180919] LNet: 53117:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1006.292540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.305920] LNet: 53135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.308830] LNetError: 53135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1006.311893] LNet: Removed LNI 10.1.2.3@tcp [ 1006.315450] LNetError: 53135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.325849] LNet: Removed LNI 192.168.201.33@tcp [ 1006.362849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.495815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.520641] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1006.526245] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1006.529239] LNet: 53143:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1006.759950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.795419] LNet: 53169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.803178] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1006.818471] LNet: Removed LNI 10.1.2.3@tcp [ 1006.821968] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.848317] LNet: Removed LNI 192.168.201.33@tcp [ 1006.878433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.929488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.944405] LNet: Added LNI 192.168.201.33@tcp [8/256/0/182] [ 1006.948064] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1007.033595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.045606] LNet: 53197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.050042] LNetError: 53197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.060368] LNet: Removed LNI 10.1.2.3@tcp [ 1007.061512] LNetError: 53197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.071431] LNet: Removed LNI 192.168.201.33@tcp [ 1007.091686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.170602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.209083] LNet: Added LNI 192.168.201.33@tcp [8/256/0/182] [ 1007.219468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1007.224280] LNet: 53205:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1007.304603] LNet: Removed LNI 10.1.2.3@tcp [ 1007.326260] LNet: Removed LNI 192.168.201.33@tcp [ 1008.021084] Key type .llcrypt unregistered [ 1008.028815] Key type ._llcrypt unregistered [ 1020.724876] Key type ._llcrypt registered [ 1020.727290] Key type .llcrypt registered [ 1021.929837] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 22:21:08 (1789266068) [ 1023.917315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1025.838231] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 22:21:12 (1789266072) [ 1026.431706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.463371] alg: No test for adler32 (adler32-zlib) [ 1027.481965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.540903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1027.577993] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1028.548175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.33 [ 1028.571659] LNet: 54467:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1028.581840] LNet: Removed LNI 192.168.201.33@tcp [ 1035.624303] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 22:21:22 (1789266082) [ 1035.752370] LNet: 55031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.759430] LNetError: 55031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.784478] LNet: Removed LNI 192.168.201.33@tcp [ 1036.386597] Key type .llcrypt unregistered [ 1036.389278] Key type ._llcrypt unregistered [ 1043.527482] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1043.795810] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1046.666882] Key type ._llcrypt registered [ 1046.668708] Key type .llcrypt registered [ 1046.946256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1046.983785] alg: No test for adler32 (adler32-zlib) [ 1048.433714] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1053.681234] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 22:21:40 (1789266100) [ 1053.755801] LNet: 55939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.764984] LNetError: 55939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1053.794832] LNet: Removed LNI 10.1.2.3@tcp [ 1054.311528] Key type .llcrypt unregistered [ 1054.313437] Key type ._llcrypt unregistered [ 1060.499865] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1060.757411] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1060.762202] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1060.766824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1063.581537] Key type ._llcrypt registered [ 1063.586552] Key type .llcrypt registered [ 1063.939847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1063.966053] alg: No test for adler32 (adler32-zlib) [ 1064.958378] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1071.910552] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 22:21:58 (1789266118) [ 1073.207198] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1074.723906] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 22:22:01 (1789266121) [ 1074.884463] LNet: 57125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.890786] LNetError: 57125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1074.918273] LNet: Removed LNI 10.1.2.3@tcp [ 1075.542959] Key type .llcrypt unregistered [ 1075.544574] Key type ._llcrypt unregistered [ 1082.200594] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1082.466994] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1082.474940] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1082.478479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1085.390114] Key type ._llcrypt registered [ 1085.393035] Key type .llcrypt registered [ 1085.618558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1085.640426] alg: No test for adler32 (adler32-zlib) [ 1086.582547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.668764] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1094.795524] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 22:22:21 (1789266141) [ 1095.214500] LNet: 58355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.222042] LNetError: 58355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1095.250745] LNet: Removed LNI 10.1.2.3@tcp [ 1096.288686] Key type .llcrypt unregistered [ 1096.293078] Key type ._llcrypt unregistered [ 1103.952473] Key type ._llcrypt registered [ 1103.954511] Key type .llcrypt registered [ 1104.202605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1104.221805] alg: No test for adler32 (adler32-zlib) [ 1105.145455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1105.171694] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1113.081733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 1113.725759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_interrupt -r 1 [ 1113.759294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_interrupt -r 1 [ 1113.815634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1113.861353] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1114.138867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.185264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1114.218439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.252935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.279750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.298821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.324394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.356328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.382878] LNet: 58790:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.398884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.400369] LNet: 1 local NIs in recovery (showing 1): 192.168.201.33@tcp [ 1114.410105] LNet: There was an unexpected network error while writing to 192.168.201.33: rc = -22 [ 1114.439273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.469299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.520240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.544358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.562905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.601258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.069411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_dropped -r 1 [ 1115.115497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_dropped -r 1 [ 1115.141488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1115.422841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.458430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1115.494321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.528740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1115.571556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.613436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.655794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.722515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.747757] LNet: 58900:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.761741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.781412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.807536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.833419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.858903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.885231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.912533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1116.451884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_aborted -r 1 [ 1116.501548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_aborted -r 1 [ 1116.564400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1116.605274] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1116.613908] LNet: Skipped 1 previous similar message [ 1116.867117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.887861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1116.919427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.952333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.002983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.064087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.119412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.164371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.189098] LNet: 59009:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1117.221204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.248164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1117.266647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.292577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.314941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.335477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.351985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1117.913197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_no_route -r 1 [ 1117.951586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_no_route -r 1 [ 1117.981960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1118.220605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.256257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1118.329043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.374459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.422991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.464651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.520154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.578570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.647826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.698924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.725614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.754243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.786387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.834141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.872866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1119.535804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_timeout -r 1 [ 1119.573267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_timeout -r 1 [ 1119.613541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1119.642935] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1119.649832] LNet: Skipped 1 previous similar message [ 1119.858141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.877139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.897272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.919157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.952263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.991493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.041545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.072041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.106950] LNet: 59227:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.125451] LNet: 59227:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1120.164742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.229744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.297175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.376495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.439781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.506670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.591667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1121.259382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_error -r 1 [ 1121.297684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_error -r 1 [ 1121.325528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1121.561360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.597193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.640228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.678777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.711948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.754205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.792572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.859743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.909689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.995344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.095727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.142661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.184964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.223514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.260325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.581540] LNet: 59378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.588049] LNetError: 59378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.630427] LNet: Removed LNI 192.168.201.33@tcp [ 1124.090710] Key type .llcrypt unregistered [ 1124.094261] Key type ._llcrypt unregistered [ 1144.461801] Key type ._llcrypt registered [ 1144.464648] Key type .llcrypt registered [ 1145.610284] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 22:23:12 (1789266192) [ 1146.566395] Key type .llcrypt unregistered [ 1146.568814] Key type ._llcrypt unregistered [ 1153.528186] Key type ._llcrypt registered [ 1153.532592] Key type .llcrypt registered [ 1153.976635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1154.043606] alg: No test for adler32 (adler32-zlib) [ 1155.052753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1155.090461] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1164.972249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 1180.103837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1180.163543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1180.207953] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1181.816251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_interrupt -r 1 [ 1181.851229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_interrupt -r 1 [ 1181.887940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_interrupt -r 1 [ 1181.920511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_interrupt -r 1 [ 1181.953308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_interrupt -r 1 [ 1181.986109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_interrupt -r 1 [ 1182.022344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_interrupt -r 1 [ 1182.059933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_interrupt -r 1 [ 1182.131504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1182.158910] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1182.174454] LNet: 1 local NIs in recovery (showing 1): 192.168.201.33@tcp [ 1182.424648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.481370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.508557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.566621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.611973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.636268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.672308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.700100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.720248] LNet: 60844:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.736785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.781410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.817311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.858241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.897134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.928726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.969612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.641523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_dropped -r 1 [ 1183.696099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_dropped -r 1 [ 1183.753596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_dropped -r 1 [ 1183.787612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_dropped -r 1 [ 1183.818900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_dropped -r 1 [ 1183.859843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_dropped -r 1 [ 1183.913520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_dropped -r 1 [ 1183.950843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_dropped -r 1 [ 1183.982444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1184.009634] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1184.012016] LNet: Skipped 2 previous similar messages [ 1184.270547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.316385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.361967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.434587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.488497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.526935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.581507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.621462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.646512] LNet: 60966:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.668931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.705407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.780631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.828046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.864535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.891122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.930503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.556430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_aborted -r 1 [ 1185.598959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_aborted -r 1 [ 1185.654860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_aborted -r 1 [ 1185.700275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_aborted -r 1 [ 1185.782888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_aborted -r 1 [ 1185.821964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_aborted -r 1 [ 1185.867550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_aborted -r 1 [ 1185.920108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_aborted -r 1 [ 1186.010848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1186.041137] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1186.044014] LNet: Skipped 2 previous similar messages [ 1186.400559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.494104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.559432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.610727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.684035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.737658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.773412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.816080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.836155] LNet: 61087:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.865450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.941653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.008568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.041927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.092399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.135649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.183190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1188.004651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_no_route -r 1 [ 1188.040243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_no_route -r 1 [ 1188.107955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_no_route -r 1 [ 1188.178701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_no_route -r 1 [ 1188.267232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_no_route -r 1 [ 1188.300641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_no_route -r 1 [ 1188.326528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_no_route -r 1 [ 1188.357679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_no_route -r 1 [ 1188.388231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1188.401149] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1188.405188] LNet: Skipped 4 previous similar messages [ 1188.550724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.566407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.586679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.617304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.658869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.717816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.763554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.803893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.852712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.888810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.917984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.952538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.980418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.017548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.049964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1189.513593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_timeout -r 1 [ 1189.554664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_timeout -r 1 [ 1189.587812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_timeout -r 1 [ 1189.621948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_timeout -r 1 [ 1189.667938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_timeout -r 1 [ 1189.703514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_timeout -r 1 [ 1189.739663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_timeout -r 1 [ 1189.780408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_timeout -r 1 [ 1189.833803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1190.172991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1190.228326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1190.272510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.320630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.364080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.408345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.457091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.536268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.549084] LNet: 61329:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.555306] LNet: 61329:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1190.593875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.657673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.708907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.774114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.819401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.868444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.903531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1191.574199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_error -r 1 [ 1191.618583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_error -r 1 [ 1191.671117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_error -r 1 [ 1191.705972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_error -r 1 [ 1191.739414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_error -r 1 [ 1191.777562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_error -r 1 [ 1191.822373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_error -r 1 [ 1191.862441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_error -r 1 [ 1191.915350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1192.303488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.342765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1192.389038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.431942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.477225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.542315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.603353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.664227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.712913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.766839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.819850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.857246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.899430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.940942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.997578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1199.434531] LNet: 61527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.449664] LNetError: 61527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.455378] LNet: Removed LNI 192.168.201.33@tcp [ 1200.494354] LNet: Removed LNI 192.168.201.33@tcp1 [ 1201.036572] Key type .llcrypt unregistered [ 1201.039125] Key type ._llcrypt unregistered [ 1219.410913] Key type ._llcrypt registered [ 1219.413875] Key type .llcrypt registered [ 1220.607760] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:24:27 (1789266267) [ 1221.357346] Key type .llcrypt unregistered [ 1221.359679] Key type ._llcrypt unregistered [ 1228.117500] Key type ._llcrypt registered [ 1228.121519] Key type .llcrypt registered [ 1228.367651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1228.380950] alg: No test for adler32 (adler32-zlib) [ 1229.312228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1229.345947] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1237.060155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 1237.866772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e remote_dropped -r 1 [ 1237.923722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e remote_dropped -r 1 [ 1237.971278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1237.990803] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1238.173763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.207230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1238.230826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.253392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.279342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.308792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.331792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.355983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.371348] LNet: 62774:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.396380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.420571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.442635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.467773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.498983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.526964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.560307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1238.977991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e remote_error -r 1 [ 1239.021698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e remote_error -r 1 [ 1239.046287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1239.062546] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1239.273508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.293705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1239.308989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.328895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.369433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.396750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.457536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.491285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.516648] LNet: 62883:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.536674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.588969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.651924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.727474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.765965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.805368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.846498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1240.445616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e remote_timeout -r 1 [ 1240.501542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e remote_timeout -r 1 [ 1240.533959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1240.553814] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1240.809616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.836599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.862397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.899571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.931758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.966756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.992404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.027448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.046352] LNet: 62992:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.067439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.113179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.193316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.230851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.258985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.283441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.307692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1241.616681] LNet: 63033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.625369] LNetError: 63033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.645975] LNet: Removed LNI 192.168.201.33@tcp [ 1242.163501] Key type .llcrypt unregistered [ 1242.165646] Key type ._llcrypt unregistered [ 1262.698943] Key type ._llcrypt registered [ 1262.701961] Key type .llcrypt registered [ 1264.397761] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:25:10 (1789266310) [ 1266.011436] Key type .llcrypt unregistered [ 1266.022923] Key type ._llcrypt unregistered [ 1274.255995] Key type ._llcrypt registered [ 1274.258710] Key type .llcrypt registered [ 1274.474987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1274.493498] alg: No test for adler32 (adler32-zlib) [ 1275.481507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1275.496167] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1285.290527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 1294.585616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1294.633624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1294.676156] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1296.194259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e remote_dropped -r 1 [ 1296.258390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e remote_dropped -r 1 [ 1296.336890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e remote_dropped -r 1 [ 1296.379229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e remote_dropped -r 1 [ 1296.420909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e remote_dropped -r 1 [ 1296.458332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e remote_dropped -r 1 [ 1296.495300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e remote_dropped -r 1 [ 1296.564590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e remote_dropped -r 1 [ 1296.608706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1296.645300] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1296.654697] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.133@tcp [ 1297.079521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.148619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1297.199583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.232301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.268102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.295599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.353808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.386989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.430099] LNet: 64510:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.470799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.518329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.594241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.638816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.702476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.704576] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1297.727351] LNet: Skipped 3 previous similar messages [ 1297.769123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.804087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1297.828545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.518303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e remote_error -r 1 [ 1298.550508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e remote_error -r 1 [ 1298.571570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e remote_error -r 1 [ 1298.613465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e remote_error -r 1 [ 1298.654562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e remote_error -r 1 [ 1298.703615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e remote_error -r 1 [ 1298.751317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e remote_error -r 1 [ 1298.836643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e remote_error -r 1 [ 1298.888962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1298.908425] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1299.135281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1299.167236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1299.204207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1299.231520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1299.260627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1299.294626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1299.335988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1299.362318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1299.376847] LNet: 64633:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1299.389541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1299.414726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1299.450129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1299.499611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1299.534334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1299.565580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1299.597545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1299.624642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1300.232593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e remote_timeout -r 1 [ 1300.263549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e remote_timeout -r 1 [ 1300.348144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e remote_timeout -r 1 [ 1300.394549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e remote_timeout -r 1 [ 1300.471653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e remote_timeout -r 1 [ 1300.514706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e remote_timeout -r 1 [ 1300.596371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e remote_timeout -r 1 [ 1300.643135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e remote_timeout -r 1 [ 1300.701954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp [ 1301.142804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1301.174890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1301.199883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1301.221444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1301.242865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1301.275996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1301.307337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1301.341311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1301.360552] LNet: 64757:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1301.386592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1301.417070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1301.450879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1301.503375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1301.563895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1301.588643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1301.686506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1301.736150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.523454] LNet: 64835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1307.529458] LNetError: 64835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1307.535695] LNet: Removed LNI 192.168.201.33@tcp [ 1307.549309] LNet: Removed LNI 192.168.201.33@tcp1 [ 1307.908468] Key type .llcrypt unregistered [ 1307.910894] Key type ._llcrypt unregistered [ 1326.794394] Key type ._llcrypt registered [ 1326.798688] Key type .llcrypt registered [ 1328.055476] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:26:14 (1789266374) [ 1329.992234] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1331.499108] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:26:18 (1789266378) [ 1332.501913] Key type .llcrypt unregistered [ 1332.504912] Key type ._llcrypt unregistered [ 1339.818607] Key type ._llcrypt registered [ 1339.821937] Key type .llcrypt registered [ 1339.997627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1340.014759] alg: No test for adler32 (adler32-zlib) [ 1340.928528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1340.952952] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1348.187633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 1348.739627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e network_timeout -r 1 [ 1348.809842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e network_timeout -r 1 [ 1348.843917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 1348.864709] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1349.114607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1349.134713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1349.156407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1349.156680] LNet: 1 local NIs in recovery (showing 1): 192.168.201.33@tcp [ 1349.177984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1349.204593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1349.222638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1349.255286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1349.287634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1349.327206] LNet: 66374:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1349.346219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1349.413246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1349.464730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.521353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.566844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.625073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1350.031828] LNet: 66413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1350.043678] LNetError: 66413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.084463] LNet: Removed LNI 192.168.201.33@tcp [ 1351.568668] Key type .llcrypt unregistered [ 1351.570753] Key type ._llcrypt unregistered [ 1370.479228] Key type ._llcrypt registered [ 1370.482670] Key type .llcrypt registered [ 1370.687940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1370.711847] alg: No test for adler32 (adler32-zlib) [ 1371.613913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1371.637267] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1379.992659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 1389.475925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.505471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1389.521515] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1391.250870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e network_timeout -r 1 [ 1391.294749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e network_timeout -r 1 [ 1391.342634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e network_timeout -r 1 [ 1391.408590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e network_timeout -r 1 [ 1391.483054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e network_timeout -r 1 [ 1391.565028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e network_timeout -r 1 [ 1391.638495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e network_timeout -r 1 [ 1391.692802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e network_timeout -r 1 [ 1391.742532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 1391.789870] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1392.122741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1392.164770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1392.204555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1392.224430] LNet: 1 local NIs in recovery (showing 1): 192.168.201.33@tcp [ 1392.233807] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.133@tcp [ 1392.251416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1392.285417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1392.341713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1392.378375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1392.428589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1392.463300] LNet: 67291:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1392.483437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1392.517591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1392.562769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1392.611750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1392.652718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1392.695937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1398.091099] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1398.100585] LNet: Skipped 2 previous similar messages [ 1398.622783] LNet: 67365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.627361] LNetError: 67365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.632792] LNet: Removed LNI 192.168.201.33@tcp [ 1398.642810] LNet: Removed LNI 192.168.201.33@tcp1 [ 1398.983739] Key type .llcrypt unregistered [ 1398.985922] Key type ._llcrypt unregistered [ 1415.265482] Key type ._llcrypt registered [ 1415.267552] Key type .llcrypt registered [ 1416.458911] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 22:27:43 (1789266463) [ 1416.787984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1416.803448] alg: No test for adler32 (adler32-zlib) [ 1417.743889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1417.839194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1417.872685] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1418.515303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1418.536047] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1419.292165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 1419.356255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1419.403424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1419.462453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1419.522815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1419.568509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1419.627702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.33@tcp [ 1419.658216] LNetError: 68463:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.33@tcp to recovery [ 1419.873443] LNet: 1 local NIs in recovery (showing 1): 192.168.201.33@tcp [ 1419.873881] LNet: There was an unexpected network error while writing to 192.168.201.33: rc = -22 [ 1421.920901] LNet: There was an unexpected network error while writing to 192.168.201.33: rc = -22 [ 1422.140403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1426.028530] LNet: There was an unexpected network error while writing to 192.168.201.33: rc = -22 [ 1426.844364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1426.914346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1427.015745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1427.047500] LNet: 68571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.054375] LNetError: 68571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.064391] LNet: Removed LNI 192.168.201.33@tcp [ 1428.073947] LNetError: 68571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.103749] LNet: Removed LNI 192.168.201.33@tcp1 [ 1428.175036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.244403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1428.290451] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1429.015020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1429.042221] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1429.982531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 1430.033994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1430.104320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1430.162492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1430.191961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1430.242498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1430.281861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1430.320573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.33@tcp [ 1430.340555] LNetError: 68844:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.33@tcp to recovery [ 1431.265045] LNet: There was an unexpected network error while writing to 192.168.201.33: rc = -22 [ 1433.928798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1437.413531] LNet: There was an unexpected network error while writing to 192.168.201.33: rc = -22 [ 1437.420823] LNet: Skipped 1 previous similar message [ 1442.037390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1442.080227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1442.123413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1449.809086] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 22:28:16 (1789266496) [ 1449.889187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.938619] LNet: 69555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.949888] LNetError: 69555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.976778] LNet: Removed LNI 192.168.201.33@tcp [ 1450.988068] LNet: Removed LNI 192.168.201.33@tcp1 [ 1451.040934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.086735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1451.114338] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1451.943697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1451.973103] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1452.948749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 1453.034954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1453.069955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1453.110471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1453.162198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1453.190239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1453.227993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.33@tcp --health 0 [ 1453.318908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1454.114476] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.33@tcp [ 1454.115349] LNet: There was an unexpected network error while writing to 192.168.201.33: rc = -22 [ 1454.124195] LNet: Skipped 1 previous similar message [ 1469.124661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1469.403965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1469.481771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1469.535652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.33@tcp --health 500 [ 1469.652243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1476.585672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1476.702846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1476.759886] LNet: 70054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.779547] LNetError: 70054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.792610] LNetError: 70054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1476.808501] LNet: Removed LNI 192.168.201.33@tcp [ 1476.833354] LNet: Removed LNI 192.168.201.33@tcp1 [ 1476.908204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.000906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1477.038431] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1478.263573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1478.293349] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1479.555175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 1479.609353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1479.716665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1479.759255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1479.826951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1479.865570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1479.897437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1479.956262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.33@tcp --health 0 [ 1480.033190] LNet: There was an unexpected network error while writing to 192.168.201.33: rc = -22 [ 1480.043636] LNet: Skipped 3 previous similar messages [ 1481.393403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1490.788643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1490.833175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1490.863957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1497.541797] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 22:29:04 (1789266544) [ 1498.381962] LNet: 71088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.390263] LNetError: 71088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1498.400524] LNetError: 71088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1498.415268] LNet: Removed LNI 192.168.201.33@tcp [ 1498.435478] LNet: Removed LNI 192.168.201.33@tcp1 [ 1499.174138] Key type .llcrypt unregistered [ 1499.182365] Key type ._llcrypt unregistered [ 1506.413831] Key type ._llcrypt registered [ 1506.421322] Key type .llcrypt registered [ 1506.685165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1506.733771] alg: No test for adler32 (adler32-zlib) [ 1507.787268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1507.825582] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1517.133443] Lustre: DEBUG MARKER: Initial discovery [ 1517.357393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.133@tcp [ 1519.567836] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1519.784182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1519.856537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.133@tcp [ 1519.878384] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1519.901699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1519.921478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.133@tcp -r 1 -m GET -l 3 [ 1519.956925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.133@tcp -r 1 -m GET -e local_error [ 1519.987331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.133@tcp -r 1 -m PUT -l 6 [ 1520.160739] LNet: 1 local NIs in recovery (showing 1): 192.168.201.33@tcp [ 1520.920986] Lustre: DEBUG MARKER: Force oleg133-client.virtnet to discover 192.168.201.133@tcp (in background) [ 1521.000198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.133@tcp [ 1522.346627] Lustre: DEBUG MARKER: Force oleg133-server to discover 192.168.201.33@tcp [ 1524.064583] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1524.352485] Lustre: DEBUG MARKER: Wait for 71841 [ 1525.867897] Lustre: DEBUG MARKER: Finished wait on 71841 [ 1526.115108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1526.174230] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1526.225049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1526.658367] LNet: 72299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.665811] LNetError: 72299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.722066] LNet: Removed LNI 192.168.201.33@tcp [ 1528.143438] Key type .llcrypt unregistered [ 1528.146687] Key type ._llcrypt unregistered [ 1550.073936] Key type ._llcrypt registered [ 1550.076674] Key type .llcrypt registered [ 1551.678330] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:29:57 (1789266597) [ 1552.506946] Key type .llcrypt unregistered [ 1552.521270] Key type ._llcrypt unregistered [ 1560.851884] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1561.057082] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1561.062643] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1561.066547] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1563.917436] Key type ._llcrypt registered [ 1563.921817] Key type .llcrypt registered [ 1564.212541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1564.242654] alg: No test for adler32 (adler32-zlib) [ 1565.274701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.336951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1565.416452] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1566.770901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1566.819286] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1579.521779] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:30:26 (1789266626) [ 1579.947368] LNet: 74373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.960783] LNetError: 74373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1579.973687] LNet: Removed LNI 10.1.2.3@tcp [ 1580.009490] LNet: Removed LNI 192.168.201.33@tcp [ 1580.983335] Key type .llcrypt unregistered [ 1580.987235] Key type ._llcrypt unregistered [ 1589.146666] Key type ._llcrypt registered [ 1589.148959] Key type .llcrypt registered [ 1589.307991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1589.323341] alg: No test for adler32 (adler32-zlib) [ 1590.325205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.367149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.398097] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1591.175253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1591.189263] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1592.096634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.33@tcp [ 1593.121253] LNetError: Unexpected error -22 connecting to 192.168.201.33@tcp at host 192.168.201.33:988 [ 1598.723122] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:30:45 (1789266645) [ 1599.128334] LNet: 75501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.137335] LNetError: 75501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1599.151828] LNet: Removed LNI 10.1.2.3@tcp [ 1599.194685] LNet: Removed LNI 192.168.201.33@tcp [ 1600.075205] Key type .llcrypt unregistered [ 1600.077831] Key type ._llcrypt unregistered [ 1606.451428] Key type ._llcrypt registered [ 1606.453798] Key type .llcrypt registered [ 1606.795388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1606.828833] alg: No test for adler32 (adler32-zlib) [ 1607.776362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.859336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1607.890396] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1608.660522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1608.677401] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 1609.619501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.33@tcp1 --nid 192.168.201.33@tcp2 [ 1614.280076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp1 192.168.201.33@tcp2 [ 1614.305568] LNetError: 76335:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.33@tcp1 to 192.168.201.33@tcp2 - Route table may be misconfigured [ 1614.339926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp2 192.168.201.33@tcp1 [ 1621.541704] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:31:08 (1789266668) [ 1621.909363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.929364] LNet: 76909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.935604] LNetError: 76909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.944187] LNet: Removed LNI 192.168.201.33@tcp1 [ 1622.950594] LNetError: 76909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.977844] LNet: Removed LNI 192.168.201.33@tcp2 [ 1623.030896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.132426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.222145] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1624.094928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1624.112834] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1625.104109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 1625.215345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp 192.168.201.33@tcp [ 1625.284716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp1 192.168.201.33@tcp1 [ 1625.340870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.33@tcp -d 192.168.201.33@tcp -e network_timeout [ 1625.418144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -e network_timeout [ 1625.461929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -e network_timeout [ 1625.544814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -e network_timeout [ 1625.625949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1625.670744] LNet: There was an unexpected network error while writing to 192.168.201.33: rc = -22 [ 1625.705517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1625.866265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1625.961570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1626.156041] LNet: 1 local NIs in recovery (showing 1): 192.168.201.33@tcp [ 1632.980292] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:31:19 (1789266679) [ 1633.155906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.196376] LNet: 77767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.206813] LNetError: 77767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.222298] LNet: Removed LNI 192.168.201.33@tcp [ 1634.280223] LNet: Removed LNI 192.168.201.33@tcp1 [ 1634.344617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.458921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1634.793423] LNet: 77808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.271583] Key type .llcrypt unregistered [ 1635.274942] Key type ._llcrypt unregistered [ 1647.851433] Key type ._llcrypt registered [ 1647.854521] Key type .llcrypt registered [ 1649.363696] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:31:35 (1789266695) [ 1649.991541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1650.002446] alg: No test for adler32 (adler32-zlib) [ 1650.977705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.391676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1654.733300] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:31:40 (1789266700) [ 1654.815768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.857079] LNet: 78896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.889055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.939591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1654.961672] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1655.776148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1655.799306] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1656.807272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1656.850543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp1 [ 1656.891393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp1 [ 1663.309172] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:31:49 (1789266709) [ 1665.350106] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg133-server [ 1667.257405] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:31:53 (1789266713) [ 1669.102945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg133-server [ 1670.535502] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:31:57 (1789266717) [ 1672.158060] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg133-server [ 1673.933780] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:32:00 (1789266720) [ 1676.234258] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg133-server [ 1678.518477] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:32:04 (1789266724) [ 1681.130260] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg133-server [ 1683.286477] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:32:09 (1789266729) [ 1685.568465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg133-server [ 1688.058910] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:32:14 (1789266734) [ 1690.029983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg133-server [ 1692.112756] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:32:18 (1789266738) [ 1693.642936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg133-server [ 1696.621209] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:32:22 (1789266742) [ 1698.988524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg133-server [ 1700.548387] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:32:27 (1789266747) [ 1701.148621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.188994] LNet: 82781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.194161] LNetError: 82781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.200178] LNet: Removed LNI 192.168.201.33@tcp [ 1702.256198] LNet: Removed LNI 192.168.201.33@tcp1 [ 1702.270084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.300474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.318674] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1703.009636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1703.141980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1703.343531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.354829] LNet: 82929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.360816] LNetError: 82929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.364260] LNetError: 82929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1703.373658] LNet: Removed LNI 192.168.201.33@tcp [ 1703.391545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.434277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1703.459522] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1704.511197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1704.682472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.710680] LNetError: 83074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.766653] LNet: Removed LNI 192.168.201.33@tcp [ 1705.780045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.828904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.848475] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1706.419848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1706.553472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1706.847614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.874142] LNet: 83222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.878576] LNet: 83222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1706.884592] LNetError: 83222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.902583] LNet: Removed LNI 192.168.201.33@tcp [ 1706.922897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.985692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1707.024204] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1708.144829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1708.375468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.414828] LNet: Removed LNI 192.168.201.33@tcp [ 1709.454695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.571091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.612982] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1710.463895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1710.575541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1711.980733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.007699] LNet: 83523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.015636] LNet: 83523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1712.031949] LNetError: 83523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.034110] LNetError: 83523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1712.057841] LNet: Removed LNI 192.168.201.33@tcp [ 1712.086745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.153942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1712.248427] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1713.415616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1713.617458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.668250] LNet: Removed LNI 192.168.201.33@tcp [ 1714.728257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.889458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.911284] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1715.331404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1715.413447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1715.599253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.648352] LNet: Removed LNI 192.168.201.33@tcp [ 1716.697249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.771694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1716.791399] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1717.916209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1718.212263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.279337] LNet: Removed LNI 192.168.201.33@tcp [ 1719.336389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.408752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1719.437962] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1719.989039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1720.048942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1720.329313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.363258] LNet: 84109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.373474] LNet: 84109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1720.385408] LNetError: 84109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.401112] LNetError: 84109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1720.428435] LNet: Removed LNI 192.168.201.33@tcp [ 1720.460333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.568041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1720.619439] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1721.326416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1721.582274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.666256] LNet: Removed LNI 192.168.201.33@tcp [ 1722.721877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.772941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.809899] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1723.581853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1723.683095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1723.916901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.966109] LNet: Removed LNI 192.168.201.33@tcp [ 1724.981696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.068312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1725.105287] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1725.946350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 1727.439896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.551297] LNet: Removed LNI 192.168.201.33@tcp [ 1728.586674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.657667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1728.691317] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1729.489860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1729.550591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.575961] LNet: Removed LNI 192.168.201.33@tcp [ 1729.590291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.637920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.664042] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1730.423813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1730.487471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.515482] LNet: Removed LNI 192.168.201.33@tcp [ 1730.531881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.577164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1730.655876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.686529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.738647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1730.836513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.886781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.979609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1731.161909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.225447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.307659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1731.345774] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1732.349485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.376484] LNet: Removed LNI 192.168.201.33@tcp [ 1732.402436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.496718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1739.723727] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:33:06 (1789266786) [ 1740.086790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.108508] LNet: 85579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.114612] LNet: 85579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1740.139824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.206858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1740.256702] LNet: Added LNI 192.168.201.33@tcp231 [8/256/0/180] [ 1741.560936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.580632] LNetError: 85717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.584987] LNetError: 85717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1741.597223] LNet: Removed LNI 192.168.201.33@tcp231 [ 1741.624980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.754336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1741.822423] LNet: Added LNI 192.168.201.33@tcp231 [8/256/0/180] [ 1741.833884] LNet: 85725:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1742.444218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1742.475659] LNet: Removed LNI 192.168.201.33@tcp231 [ 1742.503697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1742.530744] LNet: Added LNI 192.168.201.33@tcp231 [8/256/0/180] [ 1743.877593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.919947] LNet: Removed LNI 192.168.201.33@tcp231 [ 1743.941889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.022611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1744.059986] LNet: Added LNI 192.168.201.33@tcp231 [8/256/0/0] [ 1744.070889] LNet: 85877:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1744.685885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1744.719661] LNet: Removed LNI 192.168.201.33@tcp231 [ 1744.740329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1744.784170] LNet: Added LNI 192.168.201.33@tcp231 [8/256/0/0] [ 1746.173150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.230633] LNet: Removed LNI 192.168.201.33@tcp231 [ 1746.265549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.372827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1746.399662] LNet: Added LNI 192.168.201.33@tcp231 [8/256/0/60] [ 1746.407783] LNet: 86029:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1746.957207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1747.002850] LNet: Removed LNI 192.168.201.33@tcp231 [ 1747.015455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1747.055812] LNet: Added LNI 192.168.201.33@tcp231 [8/256/0/60] [ 1748.727154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.774740] LNet: Removed LNI 192.168.201.33@tcp231 [ 1748.808970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.917872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1748.962760] LNet: Added LNI 192.168.201.33@tcp231 [8/256/0/180] [ 1748.968713] LNet: 86180:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1749.537200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1749.579749] LNet: Removed LNI 192.168.201.33@tcp231 [ 1749.598549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1749.622814] LNet: Added LNI 192.168.201.33@tcp231 [8/256/0/180] [ 1758.437479] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:33:24 (1789266804) [ 1758.548056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.586844] LNet: Removed LNI 192.168.201.33@tcp231 [ 1758.610430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.708614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1758.739382] LNet: Added LNI 192.168.201.33@tcp232 [8/256/0/180] [ 1759.497767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1759.523931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1759.585827] LNet: Removed LNI 192.168.201.33@tcp232 [ 1759.627635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1759.653684] LNet: Added LNI 192.168.201.33@tcp232 [8/256/0/180] [ 1760.661495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1760.713290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1760.741202] LNet: Removed LNI 192.168.201.33@tcp232 [ 1760.753462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1760.790583] LNet: Added LNI 192.168.201.33@tcp232 [8/256/0/180] [ 1761.528275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1761.564044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1761.594302] LNet: Removed LNI 192.168.201.33@tcp232 [ 1761.618212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1761.645387] LNet: Added LNI 192.168.201.33@tcp232 [8/256/0/180] [ 1762.584289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1762.620672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1762.657281] LNet: Removed LNI 192.168.201.33@tcp232 [ 1762.683580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1769.477320] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:33:35 (1789266815) [ 1770.145236] Key type .llcrypt unregistered [ 1770.148804] Key type ._llcrypt unregistered [ 1778.944680] Key type ._llcrypt registered [ 1778.947228] Key type .llcrypt registered [ 1779.305100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1779.325727] alg: No test for adler32 (adler32-zlib) [ 1780.486438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1780.554174] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1788.324511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 1797.234167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.344958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1797.412050] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1799.045282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_interrupt -r 2 [ 1799.080908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_interrupt -r 2 [ 1799.123433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_interrupt -r 2 [ 1799.167707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_interrupt -r 2 [ 1799.220477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_interrupt -r 2 [ 1799.254318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_interrupt -r 2 [ 1799.291461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_interrupt -r 2 [ 1799.341464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_interrupt -r 2 [ 1799.399761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp 192.168.201.133@tcp [ 1799.423260] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1799.431996] LNet: 1 local NIs in recovery (showing 1): 192.168.201.33@tcp [ 1799.703333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1799.735784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1799.756727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1799.794177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1799.861724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1799.897697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1799.935877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1799.968937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1800.010508] LNet: 88679:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1800.047763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1800.093657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1800.122692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1800.154657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1800.190811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1800.225874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1800.763501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_dropped -r 2 [ 1800.819544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_dropped -r 2 [ 1800.846126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_dropped -r 2 [ 1800.898909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_dropped -r 2 [ 1800.952352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_dropped -r 2 [ 1801.016390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_dropped -r 2 [ 1801.068766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_dropped -r 2 [ 1801.126463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_dropped -r 2 [ 1801.167595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp 192.168.201.133@tcp [ 1801.201994] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1801.499907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1801.543604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1801.589389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1801.620930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1801.652036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1801.712720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1801.755674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1801.789935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1801.812622] LNet: 88800:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1801.832877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1801.880946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1801.916542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1801.958952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1801.993501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1802.032646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1802.712188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_aborted -r 2 [ 1802.752519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_aborted -r 2 [ 1802.796555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_aborted -r 2 [ 1802.849667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_aborted -r 2 [ 1802.892714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_aborted -r 2 [ 1802.920911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_aborted -r 2 [ 1802.952983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_aborted -r 2 [ 1802.981851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_aborted -r 2 [ 1803.010086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp 192.168.201.133@tcp [ 1803.024392] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1803.285386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1803.319391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1803.353655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1803.372236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1803.415436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1803.475898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1803.522322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1803.559710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1803.587773] LNet: 88920:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1803.610894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1803.659536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1803.690206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1803.721815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1803.749356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1803.773581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1804.340433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_no_route -r 2 [ 1804.383327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_no_route -r 2 [ 1804.419967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_no_route -r 2 [ 1804.469498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_no_route -r 2 [ 1804.524373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_no_route -r 2 [ 1804.568580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_no_route -r 2 [ 1804.632618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_no_route -r 2 [ 1804.675401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_no_route -r 2 [ 1804.714179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp 192.168.201.133@tcp [ 1804.954957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1804.984198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1805.013689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1805.044971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1805.077796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1805.117381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1805.142838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1805.173288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1805.227460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1805.254306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1805.280964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1805.306832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1805.335769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1805.366713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1805.831578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e local_timeout -r 2 [ 1805.861825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e local_timeout -r 2 [ 1805.885565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e local_timeout -r 2 [ 1805.909595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e local_timeout -r 2 [ 1805.946764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e local_timeout -r 2 [ 1805.991781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e local_timeout -r 2 [ 1806.026774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e local_timeout -r 2 [ 1806.076106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e local_timeout -r 2 [ 1806.119031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp 192.168.201.133@tcp [ 1806.148441] LNet: There was an unexpected network error while writing to 192.168.201.133: rc = -22 [ 1806.154504] LNet: Skipped 1 previous similar message [ 1806.418969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1806.456899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1806.482457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1806.527325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1806.569367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1806.607737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1806.648989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1806.673666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1806.698228] LNet: 89160:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1806.705814] LNet: 89160:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1806.732846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1806.763970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1806.792884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1806.831170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1806.869341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1806.902774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1807.404299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp -m GET -e remote_dropped -r 2 [ 1807.436621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.133@tcp1 -m GET -e remote_dropped -r 2 [ 1807.464827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp -m GET -e remote_dropped -r 2 [ 1807.499483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp -d 192.168.201.33@tcp1 -m GET -e remote_dropped -r 2 [ 1807.531088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp -m GET -e remote_dropped -r 2 [ 1807.573914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.133@tcp1 -m GET -e remote_dropped -r 2 [ 1807.600485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp -m GET -e remote_dropped -r 2 [ 1807.625159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.33@tcp1 -d 192.168.201.33@tcp1 -m GET -e remote_dropped -r 2 [ 1807.677634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp 192.168.201.133@tcp [ 1807.695560] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.133@tcp [ 1807.893730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1807.936567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1807.997349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1808.053478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1808.084185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1808.105280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1808.134658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1808.174569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1808.215425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1808.249473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1808.293959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1808.324746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1808.360664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1808.407392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1815.118447] LNet: 89355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.145245] LNetError: 89355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.160993] LNet: Removed LNI 192.168.201.33@tcp [ 1815.177423] LNet: Removed LNI 192.168.201.33@tcp1 [ 1815.680649] Key type .llcrypt unregistered [ 1815.683561] Key type ._llcrypt unregistered [ 1834.383282] Key type ._llcrypt registered [ 1834.386317] Key type .llcrypt registered [ 1836.095150] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:34:42 (1789266882) [ 1836.677732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1836.725859] alg: No test for adler32 (adler32-zlib) [ 1837.762694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.837456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1837.879954] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1838.891925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1838.950169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.33@tcp [ 1839.021852] LNetError: 90332:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.33@tcp to recovery [ 1839.186283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.33@tcp [ 1839.332986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1846.804904] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:34:52 (1789266892) [ 1846.903057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1846.918318] LNet: 90914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.922688] LNetError: 90914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.938994] LNet: Removed LNI 192.168.201.33@tcp [ 1846.960418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.024017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1847.049585] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1848.181421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1848.215114] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 1849.294618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.33@tcp [ 1849.319217] LNetError: 91190:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.33@tcp to recovery [ 1850.080533] LNet: 1 local NIs in recovery (showing 1): 192.168.201.33@tcp [ 1850.106790] LNet: There was an unexpected network error while writing to 192.168.201.33: rc = -22 [ 1850.713705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.33@tcp [ 1860.348904] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:35:06 (1789266906) [ 1862.372602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg133-server [ 1864.396467] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:35:10 (1789266910) [ 1866.795945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg133-server [ 1868.998590] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:35:15 (1789266915) [ 1870.720362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg133-server [ 1872.160706] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:35:18 (1789266918) [ 1872.368771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.394374] LNet: 92820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.406113] LNetError: 92820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.414205] LNet: Removed LNI 192.168.201.33@tcp [ 1872.430597] LNet: Removed LNI 192.168.201.33@tcp1 [ 1872.457576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.510629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.530547] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1873.572764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1873.633530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1873.674714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1873.823398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1873.901935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1874.003298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1880.360680] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:35:26 (1789266926) [ 1880.455411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.481730] LNet: 93545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.489364] LNetError: 93545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.498111] LNetError: 93545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1880.524371] LNet: Removed LNI 192.168.201.33@tcp [ 1880.597614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.668607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1880.699178] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1887.127515] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:35:33 (1789266933) [ 1887.279633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1887.332395] LNet: 94252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.352202] LNetError: 94252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.381741] LNet: Removed LNI 192.168.201.33@tcp [ 1887.415479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.483908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1887.517832] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1895.141924] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:35:41 (1789266941) [ 1896.901352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1898.930650] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:35:45 (1789266945) [ 1899.847572] LNet: 95290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.858732] LNetError: 95290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.891992] LNet: Removed LNI 192.168.201.33@tcp [ 1900.615095] Key type .llcrypt unregistered [ 1900.622200] Key type ._llcrypt unregistered [ 1908.827216] Key type ._llcrypt registered [ 1908.830060] Key type .llcrypt registered [ 1909.169691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1909.219594] alg: No test for adler32 (adler32-zlib) [ 1910.223930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1910.248814] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1919.693668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 1926.573500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.133@tcp [ 1927.141390] LNet: 95706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.149739] LNetError: 95706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.168510] LNet: Removed LNI 192.168.201.33@tcp [ 1927.560637] Key type .llcrypt unregistered [ 1927.562703] Key type ._llcrypt unregistered [ 1940.321830] Key type ._llcrypt registered [ 1940.324430] Key type .llcrypt registered [ 1941.658417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1943.228412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1945.190857] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:36:31 (1789266991) [ 1947.022539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1949.125518] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:36:35 (1789266995) [ 1951.443430] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg133-server [ 1953.285911] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:36:39 (1789266999) [ 1955.109333] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg133-server [ 1957.244731] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:36:43 (1789267003) [ 1959.506913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1959.516992] alg: No test for adler32 (adler32-zlib) [ 1960.661821] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 1960.934749] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 1962.608188] Key type lgssc registered [ 1964.440459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1985.778794] Key type lgssc unregistered [ 1986.009904] LNet: 98879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.019959] LNetError: 98879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.034769] LNet: Removed LNI 192.168.201.33@tcp [ 1986.835741] Key type .llcrypt unregistered [ 1986.837559] Key type ._llcrypt unregistered [ 2002.468543] Key type ._llcrypt registered [ 2002.471796] Key type .llcrypt registered [ 2004.110492] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:37:30 (1789267050) [ 2004.805221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2004.832993] alg: No test for adler32 (adler32-zlib) [ 2005.814674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.878532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2005.927126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2006.000460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2006.062702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2013.116748] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:37:39 (1789267059) [ 2013.308475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2013.329504] LNet: 100291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.348643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.439823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.465575] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2014.369131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2014.393781] LNet: 100435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.398802] LNetError: 100435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.408955] LNet: Removed LNI 192.168.201.33@tcp [ 2014.421482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2014.489991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2014.515061] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2015.048532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2015.082464] LNetError: 100463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.111831] LNet: Removed LNI 192.168.201.33@tcp [ 2015.142323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2015.240087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2015.268083] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2023.412792] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:37:49 (1789267069) [ 2023.491264] LNet: 101046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.496392] LNet: 101046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2023.500660] LNetError: 101046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.514891] LNet: Removed LNI 192.168.201.33@tcp [ 2024.250176] Key type .llcrypt unregistered [ 2024.257174] Key type ._llcrypt unregistered [ 2032.783789] Key type ._llcrypt registered [ 2032.789489] Key type .llcrypt registered [ 2033.192275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2033.224064] alg: No test for adler32 (adler32-zlib) [ 2034.168825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.261711] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2034.264817] LNet: *** cfs_fail_loc=e003, val=100*** [ 2035.104076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2035.121903] LNet: *** cfs_fail_loc=e003, val=100*** [ 2035.125086] LNet: Skipped 2 previous similar messages [ 2035.131061] Ping reply received with mapping: [ 2035.134243] found metadata reply NI[192.168.201.33@tcp] [ 2035.136595] 21 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2035.139817] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2035.142638] 5a 5a 5a 5a 5a 5a 5a 5a [ 2035.287765] LNet: 101448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.292549] LNetError: 101448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.329733] LNet: Removed LNI 192.168.201.33@tcp [ 2037.170588] Key type .llcrypt unregistered [ 2037.172709] Key type ._llcrypt unregistered [ 2051.325596] Key type ._llcrypt registered [ 2051.332736] Key type .llcrypt registered [ 2053.350973] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:38:19 (1789267099) [ 2054.300399] Key type .llcrypt unregistered [ 2054.302966] Key type ._llcrypt unregistered [ 2061.967835] Key type ._llcrypt registered [ 2061.970529] Key type .llcrypt registered [ 2062.365354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2062.401161] alg: No test for adler32 (adler32-zlib) [ 2063.463730] LNetError: 102486:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2064.014556] Key type .llcrypt unregistered [ 2064.020193] Key type ._llcrypt unregistered [ 2069.942806] Key type ._llcrypt registered [ 2069.944713] Key type .llcrypt registered [ 2071.339929] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:38:37 (1789267117) [ 2072.094325] Key type .llcrypt unregistered [ 2072.095710] Key type ._llcrypt unregistered [ 2080.461190] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2080.468338] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2083.310958] Key type ._llcrypt registered [ 2083.313952] Key type .llcrypt registered [ 2083.495285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2083.516372] alg: No test for adler32 (adler32-zlib) [ 2084.516861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.589555] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2094.934507] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:39:01 (1789267141) [ 2095.187206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2095.209180] LNet: 104260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2095.226990] LNetError: 104260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2095.261860] LNet: Removed LNI 10.1.2.3@tcp [ 2095.308327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.147542] Key type .llcrypt unregistered [ 2096.150070] Key type ._llcrypt unregistered [ 2102.733050] Key type ._llcrypt registered [ 2102.734983] Key type .llcrypt registered [ 2102.888151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2102.900469] alg: No test for adler32 (adler32-zlib) [ 2103.802384] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2103.862469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2103.979382] LNet: 104545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.983612] LNetError: 104545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.006990] LNet: Removed LNI 192.168.201.33@tcp [ 2104.769581] Key type .llcrypt unregistered [ 2104.772289] Key type ._llcrypt unregistered [ 2116.408121] Key type ._llcrypt registered [ 2116.409424] Key type .llcrypt registered [ 2117.737365] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:39:24 (1789267164) [ 2118.491368] Key type .llcrypt unregistered [ 2118.493922] Key type ._llcrypt unregistered [ 2124.508781] Key type ._llcrypt registered [ 2124.511113] Key type .llcrypt registered [ 2124.733282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2124.750193] alg: No test for adler32 (adler32-zlib) [ 2131.688371] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:39:38 (1789267178) [ 2131.841314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.886618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.936308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2131.953254] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2132.548838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2132.575968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2132.607654] LNet: Removed LNI 192.168.201.33@tcp [ 2132.852945] LNet: 106284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.246661] Key type .llcrypt unregistered [ 2133.251992] Key type ._llcrypt unregistered [ 2146.762946] Key type ._llcrypt registered [ 2146.766575] Key type .llcrypt registered [ 2148.243534] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:39:54 (1789267194) [ 2148.676562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.700970] alg: No test for adler32 (adler32-zlib) [ 2149.621646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.688237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2149.720746] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2150.514867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2150.560180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.33@tcp [ 2150.598207] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2150.606459] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.610777] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 0 [ 2150.618371] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2150.628959] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2150.633500] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.643878] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 0 [ 2150.657536] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2150.659377] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2150.673313] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.680703] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 2200 [ 2150.689730] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2150.698031] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2150.706530] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.715849] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 2200 [ 2150.726376] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2158.089180] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:40:04 (1789267204) [ 2158.616983] LNet: 107822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.626540] LNetError: 107822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.656063] LNet: Removed LNI 192.168.201.33@tcp [ 2159.501380] Key type .llcrypt unregistered [ 2159.509150] Key type ._llcrypt unregistered [ 2166.779926] Key type ._llcrypt registered [ 2166.782541] Key type .llcrypt registered [ 2167.028684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2167.043960] alg: No test for adler32 (adler32-zlib) [ 2168.019702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2168.051949] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2176.941135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 2188.390996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.456791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2188.493623] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 2189.732154] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2189.911575] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2189.916953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2189.933181] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2192.302236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2192.345686] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2205.255434] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:40:51 (1789267251) [ 2205.575250] LNet: 109328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.586494] LNetError: 109328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.591077] LNet: Removed LNI 192.168.201.33@tcp [ 2206.624925] LNet: Removed LNI 192.168.201.33@tcp1 [ 2206.639587] LNetError: 109328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2206.661106] LNetError: 109328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2206.689664] LNet: Removed LNI 10.1.2.3@tcp99 [ 2207.697862] Key type .llcrypt unregistered [ 2207.703717] Key type ._llcrypt unregistered [ 2215.635739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2215.641941] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2218.579124] Key type ._llcrypt registered [ 2218.581554] Key type .llcrypt registered [ 2218.941828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2218.986526] alg: No test for adler32 (adler32-zlib) [ 2219.995214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.068636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2220.123388] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2221.068398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2221.093843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2224.042554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2224.132682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2224.204947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2224.238061] LNetError: 110072:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2224.281459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2224.363708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2225.161922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2225.234607] LNet: 110092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.238623] LNetError: 110092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2225.243380] LNet: Removed LNI 10.1.2.3@tcp [ 2226.281632] LNet: Removed LNI 192.168.201.33@tcp [ 2227.365591] Key type .llcrypt unregistered [ 2227.367949] Key type ._llcrypt unregistered [ 2240.205692] Key type ._llcrypt registered [ 2240.208267] Key type .llcrypt registered [ 2241.490868] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:41:27 (1789267287) [ 2242.079612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.096506] alg: No test for adler32 (adler32-zlib) [ 2243.102307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.181613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2243.208047] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2244.052522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2244.183762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg133-client@tcp [ 2250.833580] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:41:37 (1789267297) [ 2250.944530] LNet: 111613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.953917] LNetError: 111613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.972115] LNet: Removed LNI 192.168.201.33@tcp [ 2251.628605] Key type .llcrypt unregistered [ 2251.631609] Key type ._llcrypt unregistered [ 2258.824516] Key type ._llcrypt registered [ 2258.826079] Key type .llcrypt registered [ 2258.986558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2258.997595] alg: No test for adler32 (adler32-zlib) [ 2259.949604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2259.973716] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2260.004184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2267.233961] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:41:54 (1789267314) [ 2268.693491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2270.639869] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:41:57 (1789267317) [ 2271.744725] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2273.755595] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:42:00 (1789267320) [ 2273.869579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.896843] LNet: 113037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.904558] LNetError: 113037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.926434] LNet: Removed LNI 192.168.201.33@tcp [ 2273.949837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.994643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2274.017343] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2274.859501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2274.908266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2283.231571] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:42:09 (1789267329) [ 2283.362696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2283.433323] LNet: 113747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.437520] LNetError: 113747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.464535] LNet: Removed LNI 192.168.201.33@tcp [ 2283.503234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.553278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2283.591733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2284.331438] Key type .llcrypt unregistered [ 2284.337832] Key type ._llcrypt unregistered [ 2297.001874] Key type ._llcrypt registered [ 2297.010050] Key type .llcrypt registered [ 2298.506761] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:42:24 (1789267344) [ 2299.078741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.098064] alg: No test for adler32 (adler32-zlib) [ 2300.063603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.137322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.176574] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2301.287795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2301.352972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2308.671197] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:42:34 (1789267354) [ 2308.778183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2308.821496] LNet: 115304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.833718] LNetError: 115304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2308.852473] LNet: Removed LNI 192.168.201.33@tcp [ 2308.910550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2308.971344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2309.015599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2315.544330] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:42:42 (1789267362) [ 2315.592861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2315.613971] LNet: 115882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.637196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2315.692834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2315.982473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2316.011815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2316.081441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2316.302692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2316.335046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2316.416929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.439594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.484911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2316.732393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2316.763832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2316.802360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2317.322290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2317.364382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2324.046466] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:42:50 (1789267370) [ 2324.325623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.378363] LNet: 116523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.386074] LNet: 116523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2324.418625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.500815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2324.521642] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2325.447840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2325.495466] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2326.659300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2326.718905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2326.917717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.935932] LNet: 116803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2326.940978] LNetError: 116803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2327.969509] LNet: Removed LNI 192.168.201.33@tcp [ 2327.972525] LNetError: 116803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.008346] LNet: Removed LNI 192.168.201.33@tcp2 [ 2328.075179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.142436] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2328.156490] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2328.163368] LNet: 116809:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2328.888809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2329.161433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.208189] LNet: Removed LNI 192.168.201.33@tcp [ 2330.211161] LNetError: 116846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2330.228137] LNetError: 116846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2330.255738] LNet: Removed LNI 192.168.201.33@tcp2 [ 2330.347829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.427872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2330.461438] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2331.213419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2331.230917] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2332.279990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2332.307333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2332.355352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2332.537380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.562333] LNet: 117130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2332.566582] LNet: 117130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2333.600146] LNet: Removed LNI 192.168.201.33@tcp [ 2334.633662] LNet: Removed LNI 192.168.201.33@tcp2 [ 2334.650497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.734980] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2334.743821] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2334.748437] LNet: 117137:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2335.369262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2335.689662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.740593] LNetError: 117174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.751551] LNetError: 117174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2335.768097] LNet: Removed LNI 192.168.201.33@tcp [ 2336.810076] LNet: Removed LNI 192.168.201.33@tcp2 [ 2336.847876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.899469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2336.960650] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2337.735701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2337.756583] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2338.565880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2338.640625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2338.902889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.936789] LNet: Removed LNI 192.168.201.33@tcp [ 2339.945640] LNet: Removed LNI 192.168.201.33@tcp2 [ 2340.025315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.107495] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2340.111960] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2340.116429] LNet: 117459:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2340.953640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2341.273688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.301743] LNet: 117497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2341.308929] LNet: 117497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2342.368155] LNet: Removed LNI 192.168.201.33@tcp [ 2342.380273] LNet: Removed LNI 192.168.201.33@tcp2 [ 2342.450668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.576443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2342.617785] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2343.548907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2343.581558] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2344.665851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2344.721593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2344.798261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2345.094410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.145079] LNetError: 117781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2345.154487] LNetError: 117781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2345.166326] LNet: Removed LNI 192.168.201.33@tcp [ 2346.215351] LNet: Removed LNI 192.168.201.33@tcp2 [ 2346.310405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.416803] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2346.421696] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2346.427330] LNet: 117787:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.090532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2347.352244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.381567] LNet: Removed LNI 192.168.201.33@tcp [ 2348.457790] LNet: Removed LNI 192.168.201.33@tcp2 [ 2348.525640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.593386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2348.624175] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2349.388752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2349.408239] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2350.154491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2350.188971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2350.250920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2350.595504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.648161] LNet: Removed LNI 192.168.201.33@tcp [ 2351.661268] LNet: Removed LNI 192.168.201.33@tcp2 [ 2351.694270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.803386] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2351.821460] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2351.829501] LNet: 118127:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2352.461100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2352.754867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2353.824252] LNet: Removed LNI 192.168.201.33@tcp [ 2353.838503] LNet: Removed LNI 192.168.201.33@tcp2 [ 2353.874907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.973339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2354.002047] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2354.833529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2354.876949] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2355.710477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2355.757594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2355.838739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2355.974797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2356.596376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.664936] LNet: Removed LNI 192.168.201.33@tcp [ 2358.702817] LNet: Removed LNI 192.168.201.33@tcp2 [ 2358.739402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.834106] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2358.839338] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2359.499167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2368.466246] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2370.673114] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:43:36 (1789267416) [ 2370.763902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2370.811874] LNet: 119240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2370.828174] LNet: 119240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2370.840529] LNetError: 119240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2370.845118] LNetError: 119240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2370.877060] LNet: Removed LNI 192.168.201.33@tcp [ 2371.950135] LNet: Removed LNI 192.168.201.33@tcp2 [ 2371.983038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.411857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2372.437683] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2373.407867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2373.437662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2374.532454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.559492] LNet: Removed LNI 10.1.2.3@tcp [ 2374.591562] LNet: Removed LNI 192.168.201.33@tcp [ 2374.624036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2374.707760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2374.735542] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2374.747296] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2375.339612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2375.361987] LNet: Removed LNI 192.168.201.33@tcp [ 2375.371987] LNet: Removed LNI 10.1.2.3@tcp [ 2383.164715] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:43:49 (1789267429) [ 2383.345465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2383.438562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.507080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2383.540270] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2384.395262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2384.438778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2384.471793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2384.515431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2384.574858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2384.605681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2384.642773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2384.679328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2384.707161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2384.745911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2384.791309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp [ 2384.938785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2391.948109] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:43:58 (1789267438) [ 2392.103613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2393.196564] LNet: Removed LNI 192.168.201.33@tcp [ 2393.253795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2393.373891] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2393.577913] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2393.581521] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2393.601013] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2396.078049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2396.124863] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2397.314371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2397.343611] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2399.717404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2399.794620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2399.889617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.33@tcp -m PUT -e local_timeout -r 1 [ 2400.089651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2400.193577] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2400.202149] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2400.489847] LNetError: 120928:0:(lib-move.c:4801:lnet_parse()) 192.168.201.33@tcp, src 192.168.201.33@tcp: Dropping PUT (error -108 looking up sender) [ 2401.539059] LNet: Removed LNI 10.1.2.3@tcp [ 2402.616455] LNet: Removed LNI 192.168.201.33@tcp [ 2403.095981] Key type .llcrypt unregistered [ 2403.098528] Key type ._llcrypt unregistered [ 2417.851918] Key type ._llcrypt registered [ 2417.854571] Key type .llcrypt registered [ 2419.326637] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:44:25 (1789267465) [ 2419.744225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.762758] alg: No test for adler32 (adler32-zlib) [ 2420.680558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.094332] LNet: 122236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.715508] Key type .llcrypt unregistered [ 2421.716953] Key type ._llcrypt unregistered [ 2428.557563] Key type ._llcrypt registered [ 2428.560422] Key type .llcrypt registered [ 2428.765418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2428.782708] alg: No test for adler32 (adler32-zlib) [ 2429.741538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2429.776642] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2438.007992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 2438.265527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2438.948895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.133@tcp [ 2439.009019] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.133@tcp [ 2439.209180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2439.264385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2439.306261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2439.393659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2439.459148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2439.515766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2439.580144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2439.643829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2439.666431] LNet: 122683:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2439.692746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2439.753702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2439.803213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2439.851861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2439.887392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2439.976876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2440.862532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2441.168041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2441.200274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2441.250698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2441.289899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2441.335422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2441.362667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2441.390440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2441.458794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2441.484628] LNet: 122801:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2441.505714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2441.533110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2441.561931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2441.601987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2441.632393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2441.681167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2442.529240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.133@tcp [ 2442.834992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2442.862790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2442.902484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2442.931466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2442.968566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2442.994593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2443.027812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2443.058973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2443.075324] LNet: 122918:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2443.091642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2443.120726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2443.155325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2443.187079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2443.215304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2443.241652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2444.121390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2444.433061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2444.496590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2444.558431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2444.603744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2444.690973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2444.750832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2444.819618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2444.879755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2444.954890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2445.010948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2445.061471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2445.110987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2445.144872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2445.186813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2445.538689] LNet: 123081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.545938] LNetError: 123081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2446.566620] LNet: Removed LNI 192.168.201.33@tcp [ 2447.230709] Key type .llcrypt unregistered [ 2447.237787] Key type ._llcrypt unregistered [ 2466.248639] Key type ._llcrypt registered [ 2466.250349] Key type .llcrypt registered [ 2467.790748] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:45:14 (1789267514) [ 2468.235219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2468.250476] alg: No test for adler32 (adler32-zlib) [ 2469.271486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2469.911240] LNet: 123967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2470.748449] Key type .llcrypt unregistered [ 2470.754775] Key type ._llcrypt unregistered [ 2478.688745] Key type ._llcrypt registered [ 2478.691357] Key type .llcrypt registered [ 2479.033335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2479.065623] alg: No test for adler32 (adler32-zlib) [ 2480.001177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2480.023054] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2490.050548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp [ 2501.509355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2501.558390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2501.596488] LNet: Added LNI 192.168.201.33@tcp1 [8/256/0/180] [ 2502.805625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2503.498370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.133@tcp [ 2503.588108] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.133@tcp [ 2503.741854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2503.777578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2503.815508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2503.853495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2503.894796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2503.913751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2503.941663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2503.961001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2503.971905] LNet: 124619:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2503.984609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2504.007119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2504.030601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2504.054677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2504.086683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2504.125927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2505.033835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2505.396613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2505.434877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2505.472706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2505.498459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2505.523375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2505.551897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2505.576983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2505.602434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2505.617771] LNet: 124736:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2505.634307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2505.652757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2505.668230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2505.689377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2505.717957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2505.755804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2506.669320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.133@tcp [ 2506.988618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2507.028537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2507.065913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2507.112584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2507.158546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2507.187819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2507.236264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2507.289180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2507.317903] LNet: 124853:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2507.334929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2507.371568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2507.399387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2507.432627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2507.473283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2507.499513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2508.227603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2508.505686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2508.542587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2508.585731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2508.623908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2508.668135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2508.723708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2508.795722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2508.862748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2508.925679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2508.959819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2509.021439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2509.071188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2509.140091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2509.189687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2515.109164] LNet: 125052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2515.113157] LNetError: 125052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2515.119282] LNet: Removed LNI 192.168.201.33@tcp [ 2515.142810] LNet: Removed LNI 192.168.201.33@tcp1 [ 2515.633679] Key type .llcrypt unregistered [ 2515.635438] Key type ._llcrypt unregistered [ 2535.106143] Key type ._llcrypt registered [ 2535.108548] Key type .llcrypt registered [ 2536.592847] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 22:46:22 (1789267582) [ 2537.057706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2537.067637] alg: No test for adler32 (adler32-zlib) [ 2538.067395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2538.127814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2538.149911] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2539.128264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2539.150724] LNet: Added LNI 192.168.201.33@tcp2 [8/256/0/180] [ 2540.056203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp [ 2540.132750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2541.162944] LNet: Removed LNI 192.168.201.33@tcp [ 2548.279558] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 22:46:34 (1789267594) [ 2550.024153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2552.078088] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:46:38 (1789267598) [ 2559.678374] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:46:45 (1789267605) [ 2568.140928] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:46:54 (1789267614) [ 2576.111291] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:47:02 (1789267622) [ 2585.292289] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:47:11 (1789267631) [ 2594.127845] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:47:20 (1789267640) [ 2601.687184] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:47:27 (1789267647) [ 2608.340515] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:47:35 (1789267655) [ 2616.478943] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:47:42 (1789267662) [ 2627.019853] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:47:53 (1789267673) [ 2635.778623] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:48:01 (1789267681) [ 2644.889762] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:48:10 (1789267690) [ 2653.591708] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:48:19 (1789267699) [ 2662.345357] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:48:28 (1789267708) [ 2671.654652] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:48:37 (1789267717) [ 2681.006343] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:48:47 (1789267727) [ 2690.511457] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:48:56 (1789267736) [ 2698.906617] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:49:05 (1789267745) [ 2699.098267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2699.179912] LNet: 136749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2699.196364] LNetError: 136749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2699.219764] LNet: Removed LNI 192.168.201.33@tcp2 [ 2699.269378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2699.432365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2699.476616] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2701.144549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2701.190159] LNet: 136893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.197728] LNetError: 136893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2701.211766] LNet: Removed LNI 192.168.201.33@tcp [ 2701.304759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2701.400792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2701.433247] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2710.017165] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:49:16 (1789267756) [ 2718.966937] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:49:25 (1789267765) [ 2727.325327] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:49:33 (1789267773) [ 2735.202869] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:49:41 (1789267781) [ 2743.050124] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:49:49 (1789267789) [ 2751.334558] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:49:57 (1789267797) [ 2759.988698] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:50:06 (1789267806) [ 2768.245437] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:50:14 (1789267814) [ 2775.894507] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:50:22 (1789267822) [ 2783.686529] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:50:30 (1789267830) [ 2792.341908] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:50:38 (1789267838) [ 2800.122826] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:50:46 (1789267846) [ 2807.369361] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2560 sec ========== 22:50:53 (1789267853) [ 2807.652767] LNet: 144358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2807.657556] LNetError: 144358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2807.671874] LNet: Removed LNI 192.168.201.33@tcp [ 2808.596829] Key type .llcrypt unregistered [ 2808.599493] Key type ._llcrypt unregistered [ 2822.651407] Key type ._llcrypt registered [ 2822.656387] Key type .llcrypt registered [ 2823.355293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2823.365459] alg: No test for adler32 (adler32-zlib) [ 2824.600457] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 2825.011366] LNet: Added LNI 192.168.201.33@tcp [8/256/0/180] [ 2826.792936] Key type lgssc registered [ 2828.554909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2898.358059] Lustre: Mounted lustre-client - version 2.17.57_106_gd81f50d [ 2904.085129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2913.414310] Lustre: Unmounted lustre-client [ 2948.513846] Key type lgssc unregistered [ 2948.805144] LNet: 147309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2948.811978] LNetError: 147309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2948.844734] LNet: Removed LNI 192.168.201.33@tcp [ 2949.663150] Key type .llcrypt unregistered [ 2949.666225] Key type ._llcrypt unregistered