[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488218688 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002503] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009007] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.022416] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027013] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] *** VALIDATE tmpfs *** [ 0.032335] *** VALIDATE proc *** [ 0.033262] *** VALIDATE cgroup *** [ 0.035005] *** VALIDATE cgroup2 *** [ 0.036300] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043529] debug: unmapping init [mem 0xffffffff88059000-0xffffffff88060fff] [ 0.045189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046741] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054320] rcu: Hierarchical SRCU implementation. [ 0.056481] smp: Bringing up secondary CPUs ... [ 0.057616] x86: Booting SMP configuration: [ 0.058037] .... node #0, CPUs: #1 #2 #3 [ 0.061474] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.156000] node 0 deferred pages initialised in 89ms [ 0.159011] devtmpfs: initialized [ 0.160246] x86/mm: Memory block size: 128MB [ 0.162858] gcov: version magic: 0x41383552 [ 0.164139] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166288] pinctrl core: initialized pinctrl subsystem [ 0.167252] [ 0.167860] ************************************************************* [ 0.168014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169017] ** ** [ 0.170015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171017] ** ** [ 0.172016] ** This means that this kernel is built to expose internal ** [ 0.173014] ** IOMMU data structures, which may compromise security on ** [ 0.174021] ** your system. ** [ 0.175014] ** ** [ 0.176015] ** If you see this message and you are not debugging the ** [ 0.177018] ** kernel, report this immediately to your vendor! ** [ 0.178013] ** ** [ 0.179015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180017] ************************************************************* [ 0.181589] NET: Registered protocol family 16 [ 0.182476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.185494] cpuidle: using governor menu [ 0.187051] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190477] PCI: Using configuration type 1 for base access [ 0.191118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200062] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207041] cryptd: max_cpu_qlen set to 1000 [ 0.210224] ACPI: Added _OSI(Module Device) [ 0.211011] ACPI: Added _OSI(Processor Device) [ 0.212015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.217512] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.219559] ACPI: Interpreter enabled [ 0.220069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221013] ACPI: Using IOAPIC for interrupt routing [ 0.222113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223349] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233373] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236061] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238299] acpiphp: Slot [2] registered [ 0.239112] acpiphp: Slot [5] registered [ 0.240084] acpiphp: Slot [6] registered [ 0.241119] acpiphp: Slot [3] registered [ 0.243120] acpiphp: Slot [4] registered [ 0.244106] acpiphp: Slot [7] registered [ 0.246118] acpiphp: Slot [8] registered [ 0.247099] acpiphp: Slot [9] registered [ 0.249102] acpiphp: Slot [10] registered [ 0.250108] acpiphp: Slot [11] registered [ 0.252142] acpiphp: Slot [12] registered [ 0.254106] acpiphp: Slot [13] registered [ 0.255156] acpiphp: Slot [14] registered [ 0.257115] acpiphp: Slot [15] registered [ 0.258116] acpiphp: Slot [16] registered [ 0.260111] acpiphp: Slot [17] registered [ 0.262189] acpiphp: Slot [18] registered [ 0.263105] acpiphp: Slot [19] registered [ 0.265119] acpiphp: Slot [20] registered [ 0.266098] acpiphp: Slot [21] registered [ 0.268100] acpiphp: Slot [22] registered [ 0.270121] acpiphp: Slot [23] registered [ 0.271100] acpiphp: Slot [24] registered [ 0.272106] acpiphp: Slot [25] registered [ 0.274092] acpiphp: Slot [26] registered [ 0.275111] acpiphp: Slot [27] registered [ 0.277079] acpiphp: Slot [28] registered [ 0.278112] acpiphp: Slot [29] registered [ 0.280101] acpiphp: Slot [30] registered [ 0.281134] acpiphp: Slot [31] registered [ 0.283076] PCI host bridge to bus 0000:00 [ 0.285025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.292025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.294023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.297027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.302066] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306188] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.314016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.318053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.324015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.326017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.330150] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.332837] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.336043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.338929] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.343982] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.353018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.358016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.363619] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.375019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.384019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.402118] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.411892] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.422018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.427014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.438019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.448242] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.451404] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.454391] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.457383] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.459218] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.465125] iommu: Default domain type: Passthrough [ 0.467557] SCSI subsystem initialized [ 0.469130] ACPI: bus type USB registered [ 0.470180] usbcore: registered new interface driver usbfs [ 0.473084] usbcore: registered new interface driver hub [ 0.475079] usbcore: registered new device driver usb [ 0.477187] pps_core: LinuxPPS API ver. 1 registered [ 0.479014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.482067] PTP clock support registered [ 0.484101] EDAC MC: Ver: 3.0.0 [ 0.486434] PCI: Using ACPI for IRQ routing [ 0.487391] NetLabel: Initializing [ 0.488000] NetLabel: domain hash size = 128 [ 0.490009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.491057] NetLabel: unlabeled traffic allowed by default [ 0.492134] vgaarb: loaded [ 0.493179] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.494000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.502865] clocksource: Switched to clocksource kvm-clock [ 0.601740] VFS: Disk quotas dquot_6.6.0 [ 0.602658] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.604021] *** VALIDATE ramfs *** [ 0.604797] *** VALIDATE hugetlbfs *** [ 0.606736] pnp: PnP ACPI init [ 0.609049] pnp: PnP ACPI: found 6 devices [ 0.636079] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.639363] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.641585] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.643834] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.646294] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.648483] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.650904] NET: Registered protocol family 2 [ 0.653062] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.657844] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.660873] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.665569] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.668269] TCP: Hash tables configured (established 65536 bind 65536) [ 0.670961] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.673989] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.676841] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.679705] NET: Registered protocol family 1 [ 0.681695] RPC: Registered named UNIX socket transport module. [ 0.683667] RPC: Registered udp transport module. [ 0.684922] RPC: Registered tcp transport module. [ 0.686272] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.688050] NET: Registered protocol family 44 [ 0.689473] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.691027] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.692612] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.694252] PCI: CLS 0 bytes, default 64 [ 0.695518] Unpacking initramfs... [ 2.083665] debug: unmapping init [mem 0xffff8a46fcc64000-0xffff8a46fffcffff] [ 2.086756] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.088331] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.090790] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.615558] Initialise system trusted keyrings [ 2.617463] Key type blacklist registered [ 2.619701] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.628771] zbud: loaded [ 2.631707] *** VALIDATE nfs *** [ 2.633070] *** VALIDATE nfs4 *** [ 2.635268] pstore: using deflate compression [ 2.639275] Platform Keyring initialized [ 2.732735] NET: Registered protocol family 38 [ 2.734439] Key type asymmetric registered [ 2.735330] Asymmetric key parser 'x509' registered [ 2.736389] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.738310] io scheduler mq-deadline registered [ 2.739262] io scheduler kyber registered [ 2.740615] io scheduler bfq registered [ 2.742501] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.745703] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.747680] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.749577] ACPI: Power Button [PWRF] [ 2.753794] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.758139] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.765078] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.791683] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.817924] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.822025] Non-volatile memory driver v1.3 [ 2.823125] Linux agpgart interface v0.103 [ 2.853339] virtio_blk virtio1: [vda] 146536 512-byte logical blocks (75.0 MB/71.6 MiB) [ 2.855911] vda: detected capacity change from 0 to 75026432 [ 2.870407] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.873464] vdb: detected capacity change from 0 to 1073741824 [ 2.879772] libphy: Fixed MDIO Bus: probed [ 2.888703] usbcore: registered new interface driver usbserial_generic [ 2.891037] usbserial: USB Serial support registered for generic [ 2.893112] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.896655] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.898000] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.899795] mousedev: PS/2 mouse device common for all mice [ 2.901898] rtc_cmos 00:05: RTC can wake from S4 [ 2.904312] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.905489] rtc_cmos 00:05: registered as rtc0 [ 2.910260] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.910370] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.913206] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.915222] intel_pstate: CPU model not supported [ 2.920203] hid: raw HID events driver (C) Jiri Kosina [ 2.922195] usbcore: registered new interface driver usbhid [ 2.924238] usbhid: USB HID core driver [ 2.925830] drop_monitor: Initializing network drop monitor service [ 2.927870] Initializing XFRM netlink socket [ 2.929717] NET: Registered protocol family 10 [ 2.933196] Segment Routing with IPv6 [ 2.934470] NET: Registered protocol family 17 [ 2.935967] mpls_gso: MPLS GSO support [ 2.939392] RAS: Correctable Errors collector initialized. [ 2.940601] AVX version of gcm_enc/dec engaged. [ 2.941419] AES CTR mode by8 optimization enabled [ 3.010177] sched_clock: Marking stable (3010151304, 0)->(4001381572, -991230268) [ 3.012537] registered taskstats version 1 [ 3.013692] Loading compiled-in X.509 certificates [ 3.015452] zswap: loaded using pool lzo/zbud [ 3.034573] Key type big_key registered [ 3.043622] Key type encrypted registered [ 3.044633] ima: No TPM chip found, activating TPM-bypass! [ 3.045819] ima: Allocated hash algorithm: sha1 [ 3.046852] ima: No architecture policies found [ 3.047864] evm: Initialising EVM extended attributes: [ 3.048886] evm: security.selinux [ 3.049644] evm: security.ima [ 3.050268] evm: security.capability [ 3.050934] evm: HMAC attrs: 0x1 [ 3.054068] rtc_cmos 00:05: setting system clock to 2026-08-22 03:00:45 UTC (1787367645) [ 3.058531] debug: unmapping init [mem 0xffffffff89003000-0xffffffff891fffff] [ 3.060258] debug: unmapping init [mem 0xffffffff87d82000-0xffffffff88058fff] [ 3.068082] Write protecting the kernel read-only data: 28672k [ 3.070056] debug: unmapping init [mem 0xffffffff86403000-0xffffffff865fffff] [ 3.071554] debug: unmapping init [mem 0xffffffff86d14000-0xffffffff86dfffff] [ 3.097361] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.104568] systemd[1]: Detected virtualization kvm. [ 3.106713] systemd[1]: Detected architecture x86-64. [ 3.108851] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.136676] systemd[1]: No hostname configured. [ 3.138775] systemd[1]: Set hostname to . [ 3.140862] random: systemd: uninitialized urandom read (16 bytes read) [ 3.142995] systemd[1]: Initializing machine ID from random generator. [ 3.188882] random: ln: uninitialized urandom read (6 bytes read) [ 3.259153] random: systemd: uninitialized urandom read (16 bytes read) [ 3.260695] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.263777] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.267879] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.804206] device-mapper: uevent: version 1.0.3 [ 3.806137] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.363060] virtio_net virtio0 ens2: renamed from eth0 [ 4.408582] scsi host0: ata_piix [ 4.428533] scsi host1: ata_piix [ 4.430065] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.432481] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.019059] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.555863] random: crng init done [ 9.557088] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.740341] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.751807] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.979279] SELinux: Disabled at runtime. [ 11.039953] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.049264] systemd[1]: Detected virtualization kvm. [ 11.051435] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.539345] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.541739] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.545760] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.548401] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.550570] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.555784] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.562614] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [[ 11.604110] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 11.955651] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.201718] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.229391] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.334632] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.350826] EDAC sbridge: Ver: 1.1.2 [ 13.372151] Key type dns_resolver registered [ 13.684158] NFS: Registering the id_resolver key type [ 13.686306] Key type id_resolver registered [ 13.687885] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg259-client login: [ 65.293809] libcfs: loading out-of-tree module taints kernel. [ 65.344166] Key type ._llcrypt registered [ 65.345938] Key type .llcrypt registered [ 66.139130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 66.159148] alg: No test for adler32 (adler32-zlib) [ 67.913592] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 68.652671] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 70.361039] Key type lgssc registered [ 71.849734] hrtimer: interrupt took 2978389 ns [ 73.416598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 211.951781] Lustre: Mounted lustre-client - version 2.17.57_45_gb800aec [ 217.713487] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.124567] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing check_logdir /tmp/testlogs/ [ 237.542022] Lustre: lustre-OST0000-osc-ffff8a4752f72800: disconnect after 23s idle [ 238.220865] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing yml_node [ 245.246840] Lustre: DEBUG MARKER: Client: 2.17.57.45 [ 248.001978] Lustre: DEBUG MARKER: MDS: 2.17.57.45 [ 252.200275] Lustre: DEBUG MARKER: OSS: 2.17.57.45 [ 254.733726] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 21 23:04:54 EDT 2026 [ 272.808435] Lustre: Unmounted lustre-client [ 295.899668] Key type lgssc unregistered [ 296.130786] LNet: 8421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 296.135590] LNetError: 8421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 296.148893] LNet: Removed LNI 192.168.202.59@tcp [ 297.075183] Key type .llcrypt unregistered [ 297.077286] Key type ._llcrypt unregistered [ 323.695275] Key type ._llcrypt registered [ 323.698365] Key type .llcrypt registered [ 324.019656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 324.032319] alg: No test for adler32 (adler32-zlib) [ 325.021570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 325.116366] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 325.143354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 329.315613] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 329.627810] LNet: 9309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.637177] LNetError: 9309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 329.657688] LNet: Removed LNI 192.168.202.59@tcp [ 330.565639] Key type .llcrypt unregistered [ 330.569200] Key type ._llcrypt unregistered [ 336.105306] Key type ._llcrypt registered [ 336.107935] Key type .llcrypt registered [ 337.524345] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 23:06:18 (1787367978) [ 337.843200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 337.865627] alg: No test for adler32 (adler32-zlib) [ 338.750972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.818743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.836463] LNet: 9726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.857719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.913652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 338.930839] LNet: 9734:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 346.431757] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 23:06:27 (1787367987) [ 346.633425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.676169] LNet: 10313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.712622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.785029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 353.130236] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 23:06:34 (1787367994) [ 353.248357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.271806] LNet: 10891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.290292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.408394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 360.530133] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 23:06:41 (1787368001) [ 360.743432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.765796] LNet: 11484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.791835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.875936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 367.734810] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 23:06:48 (1787368008) [ 368.013307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.064397] LNet: 12068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.080702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.140423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 368.611380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 375.351534] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 23:06:56 (1787368016) [ 375.532734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.591249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.657869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 375.978864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 376.459472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 383.288632] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 23:07:04 (1787368024) [ 383.686384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.718369] LNet: 13254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.729366] LNet: 13254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 383.766466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.893427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 391.552345] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 23:07:12 (1787368032) [ 391.945824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.022502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.090127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 392.132413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 392.634765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 392.699229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 393.121734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 393.155251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 393.539329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 393.580881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 393.955820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 393.992865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.546616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 394.592070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 395.167430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 395.222811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 395.746165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 395.793982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 396.381174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 396.439174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 397.010614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 397.043702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 397.384674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 397.409461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 397.813133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 397.855758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.314453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 398.382615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 405.629924] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 23:07:26 (1787368046) [ 406.112854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.147470] LNet: 14522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.150413] LNet: 14522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 406.183867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.316752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 406.388569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 414.055937] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 23:07:34 (1787368054) [ 414.164109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.290660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.451903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 414.491826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 421.933457] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 23:07:42 (1787368062) [ 422.248629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.275769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.359394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 422.415731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 429.048269] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 23:07:50 (1787368070) [ 429.137952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.195423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.315172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 429.357056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 436.455626] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 23:07:57 (1787368077) [ 436.695743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.774578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.878967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 436.926864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 443.811376] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 23:08:04 (1787368084) [ 443.908073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.933519] LNet: 17445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.940533] LNet: 17445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 443.974196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.104353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 444.157757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 450.883499] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 23:08:11 (1787368091) [ 451.002331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.046218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.119402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 451.542567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 459.221080] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 23:08:20 (1787368100) [ 459.520171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.590540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.783810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.105817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 466.304273] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 23:08:27 (1787368107) [ 466.796183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.829687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.919452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 467.080240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.310636] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 23:08:35 (1787368115) [ 475.719720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.754426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.533890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.749724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 484.914171] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 23:08:45 (1787368125) [ 485.143794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.229698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.315626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.374265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 492.856687] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 23:08:53 (1787368133) [ 492.982968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.030147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.189977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 493.262942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.722709] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 23:09:01 (1787368141) [ 500.008323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.064658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.172411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 507.145814] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 23:09:08 (1787368148) [ 507.217294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.292701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.427907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 514.986488] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 23:09:15 (1787368155) [ 515.152310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.215922] LNet: 22886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 515.228320] LNet: 22886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 515.286097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.470206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 516.048120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 522.854938] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 23:09:23 (1787368163) [ 522.999540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.051282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.136634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 523.176599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 530.475325] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 23:09:31 (1787368171) [ 530.790898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.838321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.937865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 530.992241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 538.685240] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 23:09:39 (1787368179) [ 538.870996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.962409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.081692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 539.166322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 547.697576] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 23:09:48 (1787368188) [ 547.903752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.982429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.160974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 548.234346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 555.717838] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 23:09:56 (1787368196) [ 556.156100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.215313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.285187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 556.371623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 556.416497] LNetError: 25840:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 556.485311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 556.546530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 563.315663] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 23:10:04 (1787368204) [ 563.659874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.774018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.946776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zkmmYSEtza [ 564.056702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zkmmYSEtza [ 564.141897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zkmmYSEtza [ 571.426837] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 23:10:12 (1787368212) [ 571.511940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.571415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.656671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 571.704536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 571.745612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 578.733930] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 23:10:19 (1787368219) [ 578.855899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.919695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.994108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 579.011703] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 580.794242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 582.205395] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 584.296608] Key type lgssc registered [ 585.719786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 590.832466] LNetError: 27601:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.59@tcp99 has no available nets [ 593.399663] Key type lgssc unregistered [ 593.705836] LNetError: 28713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.725903] LNet: Removed LNI 192.168.202.59@tcp [ 595.411644] Key type .llcrypt unregistered [ 595.417183] Key type ._llcrypt unregistered [ 604.847720] Key type ._llcrypt registered [ 604.850381] Key type .llcrypt registered [ 606.871970] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 23:10:47 (1787368247) [ 607.709212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 607.754427] alg: No test for adler32 (adler32-zlib) [ 608.721950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.823264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 611.345482] LNet: Message forwarding will be enabled in 2s [ 614.432173] LNet: Message forwarding enabled [ 614.903968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 614.954337] LNet: Message forwarding will stop in 2s to 6s [ 619.833582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 620.755290] LNet: Message forwarding disabled [ 622.580604] LNet: Message forwarding will be enabled in 2s [ 625.902417] LNet: Message forwarding enabled [ 630.539332] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 23:11:11 (1787368271) [ 630.640691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.693777] LNet: 30117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.435256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.563844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 640.549289] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 23:11:21 (1787368281) [ 640.851406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.883645] LNet: 30704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.912858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.999806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 641.053982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 641.111918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 641.155832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 641.224980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 641.253444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 641.293152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 641.321401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 641.367534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 641.451558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 641.498973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 641.561778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 641.641970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 641.698741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 641.755775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 641.815801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 641.873702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 641.910565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 641.954638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 641.975516] LNetError: 30748:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 648.950380] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 23:11:29 (1787368289) [ 649.119597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.148207] LNet: 31318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.186484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.241403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 657.311403] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 23:11:38 (1787368298) [ 659.779803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg259-server [ 662.284300] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 23:11:42 (1787368302) [ 662.394773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.421021] LNet: 32244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.460860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.555314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 662.612696] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 663.459370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[60-64]@tcp [ 663.472837] LNet: 32384:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 663.536710] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 663.577695] LNetError: 32253:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.60@tcp has gone from up to down [ 664.567938] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 664.593976] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 665.577364] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 666.600937] LNetError: 32260:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.64@tcp has gone from up to down [ 667.715320] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 667.729676] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 671.783158] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 671.803300] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 672.577646] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 23:11:52 (1787368312) [ 675.509427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg259-server [ 677.822749] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 23:11:58 (1787368318) [ 678.020166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.943324] LNetError: 32252:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.202.60@tcp [ 680.095831] LNet: 33313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.111115] LNetError: 33313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 680.137853] LNet: Removed LNI 192.168.202.59@tcp [ 680.180442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.277067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 680.301262] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 681.254803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[60-65/2]@tcp [ 681.280722] LNet: 33439:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 681.296410] LNet: 33439:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 681.321789] LNetError: 33325:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 681.330386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[60-65/2]@tcp [ 681.332973] LNetError: 33325:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 681.332995] LNetError: 33325:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.60@tcp has gone from up to down [ 681.333025] LNetError: 33325:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 689.594552] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 23:12:10 (1787368330) [ 689.691843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.706832] LNet: 34027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.711291] LNetError: 34027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.727416] LNet: Removed LNI 192.168.202.59@tcp [ 689.807925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.976727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 690.057494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.115927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.205128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 690.282605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.331628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.398042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 690.519543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.577693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.663434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 690.816815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.880379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.979332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 691.148965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.234439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.362761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 691.520843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.582646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.699940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 691.833197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.917587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.032710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 692.138337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.176436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.253838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 692.352747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.387815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.449357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 692.490613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.529291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.579936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 692.648769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.702831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.759239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 700.811523] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 23:12:21 (1787368341) [ 703.107764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg259-server [ 705.300955] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 23:12:25 (1787368345) [ 707.387757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg259-server [ 709.530834] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 23:12:30 (1787368350) [ 709.657323] LNet: 35427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.659904] LNet: 35427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 710.340892] Key type .llcrypt unregistered [ 710.343406] Key type ._llcrypt unregistered [ 716.779483] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 716.972733] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 716.984352] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 716.997472] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 719.584649] Key type ._llcrypt registered [ 719.586855] Key type .llcrypt registered [ 719.752255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 719.766233] alg: No test for adler32 (adler32-zlib) [ 720.653872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.717826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 720.743065] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 721.446986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 721.460775] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 723.529769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 723.551165] LNet: Removed LNI 10.1.2.3@tcp [ 723.666633] LNet: 36158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.677831] LNetError: 36158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.695819] LNet: Removed LNI 192.168.202.59@tcp [ 724.277490] Key type .llcrypt unregistered [ 724.278636] Key type ._llcrypt unregistered [ 743.309975] Key type ._llcrypt registered [ 743.311666] Key type .llcrypt registered [ 745.656384] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 23:13:05 (1787368385) [ 746.141315] Key type .llcrypt unregistered [ 746.143409] Key type ._llcrypt unregistered [ 754.193320] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.407705] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.421461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.425959] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.412866] Key type ._llcrypt registered [ 757.418988] Key type .llcrypt registered [ 757.744464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.767910] alg: No test for adler32 (adler32-zlib) [ 758.746223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.846926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.911546] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 760.085321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 760.116506] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 764.009721] LNet: 37677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.016932] LNetError: 37677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 764.021122] LNet: Removed LNI 10.1.2.3@tcp [ 764.037955] LNet: Removed LNI 192.168.202.59@tcp [ 765.050413] Key type .llcrypt unregistered [ 765.059110] Key type ._llcrypt unregistered [ 779.667292] Key type ._llcrypt registered [ 779.670335] Key type .llcrypt registered [ 781.107694] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 23:13:41 (1787368421) [ 781.683600] Key type .llcrypt unregistered [ 781.690916] Key type ._llcrypt unregistered [ 787.167915] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 787.279314] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 787.283225] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 787.287343] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 789.977908] Key type ._llcrypt registered [ 789.981332] Key type .llcrypt registered [ 790.212318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.226992] alg: No test for adler32 (adler32-zlib) [ 791.158601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.212969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 791.249935] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 792.783822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 792.793159] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.676548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 794.704374] LNet: Removed LNI 10.1.2.3@tcp [ 794.716849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 794.757577] LNet: Removed LNI 10.1.2.31@tcp [ 794.790193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 794.829880] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.663848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 795.685752] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 796.907379] LNet: 39606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.920701] LNetError: 39606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 796.929411] LNet: Removed LNI 10.1.2.31@tcp [ 796.938647] LNet: Removed LNI 10.1.2.3@tcp [ 797.709479] Key type .llcrypt unregistered [ 797.712576] Key type ._llcrypt unregistered [ 811.120992] Key type ._llcrypt registered [ 811.122271] Key type .llcrypt registered [ 812.808151] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 23:14:13 (1787368453) [ 813.321908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 813.353290] alg: No test for adler32 (adler32-zlib) [ 814.334916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.420617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.464829] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 815.301066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.330978] LNet: 40564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.341561] LNetError: 40564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.361986] LNet: Removed LNI 192.168.202.59@tcp [ 815.376683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.499186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.59@tcp [ 815.568396] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 816.691512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.59@tcp [ 816.748058] LNet: Removed LNI 192.168.202.59@tcp [ 816.861206] LNet: 40699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.708898] Key type .llcrypt unregistered [ 817.710818] Key type ._llcrypt unregistered [ 831.666087] Key type ._llcrypt registered [ 831.667536] Key type .llcrypt registered [ 833.068447] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 23:14:33 (1787368473) [ 835.814286] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg259-server [ 838.299327] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 23:14:39 (1787368479) [ 838.811677] Key type .llcrypt unregistered [ 838.814280] Key type ._llcrypt unregistered [ 848.629483] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 848.851959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.879089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 848.881479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 850.975076] Key type ._llcrypt registered [ 850.982190] Key type .llcrypt registered [ 851.198772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 851.216526] alg: No test for adler32 (adler32-zlib) [ 852.372227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.556641] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 853.797962] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 857.481468] LNet: 42539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.486824] LNetError: 42539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.493253] LNet: Removed LNI 192.168.202.59@tcp1 [ 857.503891] LNet: Removed LNI 10.1.2.3@tcp2 [ 858.355796] Key type .llcrypt unregistered [ 858.359943] Key type ._llcrypt unregistered [ 872.705479] Key type ._llcrypt registered [ 872.707944] Key type .llcrypt registered [ 874.352570] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 23:15:14 (1787368514) [ 874.919453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 874.947885] alg: No test for adler32 (adler32-zlib) [ 875.925666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.001784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.052064] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 876.921213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.943102] LNet: 43514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.965584] LNetError: 43514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.008494] LNet: Removed LNI 192.168.202.59@tcp [ 877.043139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.117913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 877.187607] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 884.662905] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 23:15:25 (1787368525) [ 884.793455] LNet: 44096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.815561] LNetError: 44096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.844648] LNet: Removed LNI 192.168.202.59@tcp [ 885.618573] Key type .llcrypt unregistered [ 885.620302] Key type ._llcrypt unregistered [ 892.228925] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 892.508793] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 892.524170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 892.532498] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 895.240689] Key type ._llcrypt registered [ 895.243452] Key type .llcrypt registered [ 895.417836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 895.442275] alg: No test for adler32 (adler32-zlib) [ 896.440372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.541395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.579870] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 897.639409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 897.679749] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 900.399521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.437531] LNet: 44845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.473500] LNetError: 44845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 900.478735] LNet: Removed LNI 10.1.2.3@tcp [ 900.497823] LNet: Removed LNI 192.168.202.59@tcp [ 900.554150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.734041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 900.796842] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 900.800051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.754303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.788695] LNet: 44873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.792838] LNetError: 44873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.797805] LNetError: 44873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 901.814850] LNet: Removed LNI 10.1.2.3@tcp [ 901.839848] LNet: Removed LNI 192.168.202.59@tcp [ 901.883808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.027851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 902.067206] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 902.081644] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 902.935619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 902.972046] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 904.377678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.406268] LNet: 45035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.413787] LNetError: 45035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 904.422762] LNetError: 45035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 904.433335] LNet: Removed LNI 10.1.2.3@tcp [ 904.436206] LNet: Removed LNI 192.168.202.59@tcp [ 904.452941] LNet: Removed LNI 192.168.202.59@tcp2 [ 904.504767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.623405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 904.683830] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 904.688714] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 904.694685] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 912.792964] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 23:15:53 (1787368553) [ 912.915454] LNet: 45627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.928681] LNetError: 45627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 912.932725] LNetError: 45627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 912.938588] LNet: Removed LNI 10.1.2.3@tcp [ 912.944474] LNet: Removed LNI 192.168.202.59@tcp [ 912.957422] LNet: Removed LNI 192.168.202.59@tcp2 [ 913.920142] Key type .llcrypt unregistered [ 913.925152] Key type ._llcrypt unregistered [ 920.349675] Key type ._llcrypt registered [ 920.351147] Key type .llcrypt registered [ 920.643740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.669803] alg: No test for adler32 (adler32-zlib) [ 921.626160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.671501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 921.727149] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 922.767753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.790551] LNet: 46033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.795933] LNetError: 46033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.811514] LNet: Removed LNI 192.168.202.59@tcp [ 922.850610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.909880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 922.927481] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 923.610963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.642584] LNet: 46063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.648555] LNetError: 46063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.659781] LNet: Removed LNI 192.168.202.59@tcp [ 923.677981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.735112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 923.755185] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 924.229861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.303307] LNet: Removed LNI 192.168.202.59@tcp [ 924.335797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.451532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 924.474619] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 925.092613] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 925.251265] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 925.261425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 925.270155] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 927.856811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.921168] LNet: 46172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.929205] LNet: 46172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 927.940982] LNetError: 46172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.947897] LNetError: 46172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 927.971938] LNet: Removed LNI 192.168.202.59@tcp [ 928.013248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.109599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 928.143402] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 935.442122] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 23:16:16 (1787368576) [ 935.617980] LNet: 46766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.627824] LNetError: 46766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.680534] LNet: Removed LNI 192.168.202.59@tcp [ 936.467499] Key type .llcrypt unregistered [ 936.470370] Key type ._llcrypt unregistered [ 944.616994] Key type ._llcrypt registered [ 944.618806] Key type .llcrypt registered [ 944.759741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 944.768181] alg: No test for adler32 (adler32-zlib) [ 945.759977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.809510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 945.830212] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 946.730525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.753885] LNet: 47172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.759166] LNetError: 47172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.770718] LNet: Removed LNI 192.168.202.59@tcp [ 946.809395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.883396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 946.944980] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 953.805401] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 23:16:35 (1787368595) [ 953.973419] LNet: 47763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.984419] LNetError: 47763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.010841] LNet: Removed LNI 192.168.202.59@tcp [ 954.635459] Key type .llcrypt unregistered [ 954.638396] Key type ._llcrypt unregistered [ 961.705220] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 961.984180] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 961.986871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 961.991591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 964.592573] Key type ._llcrypt registered [ 964.594905] Key type .llcrypt registered [ 964.869716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.890623] alg: No test for adler32 (adler32-zlib) [ 965.924464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.973876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 966.010002] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 966.935268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.954204] LNet: 48217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.959920] LNetError: 48217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.975904] LNet: Removed LNI 192.168.202.59@tcp [ 966.993486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.056922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 967.081683] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 975.122444] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 23:16:55 (1787368615) [ 975.352551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.368552] LNet: 48816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.374416] LNetError: 48816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.394858] LNet: Removed LNI 192.168.202.59@tcp [ 975.418985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.596213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 983.001371] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 23:17:03 (1787368623) [ 983.152631] LNet: 49393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.117244] Key type .llcrypt unregistered [ 984.118751] Key type ._llcrypt unregistered [ 991.438299] Key type ._llcrypt registered [ 991.443283] Key type .llcrypt registered [ 991.743405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.766351] alg: No test for adler32 (adler32-zlib) [ 992.763325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.817838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 992.839396] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 993.832393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.868469] LNet: 49801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.876550] LNetError: 49801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.928645] LNet: Removed LNI 192.168.202.59@tcp [ 993.985423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.044669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 994.073038] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1001.892590] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 23:17:22 (1787368642) [ 1002.013569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.062815] LNet: 50389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.073720] LNetError: 50389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.114994] LNet: Removed LNI 192.168.202.59@tcp [ 1002.147702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.253376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1010.050764] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 23:17:30 (1787368650) [ 1011.755847] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1013.653904] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 23:17:34 (1787368654) [ 1015.594436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1017.916806] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 23:17:38 (1787368658) [ 1019.762503] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1021.577636] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 23:17:42 (1787368662) [ 1023.695862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1025.547851] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 23:17:46 (1787368666) [ 1027.450925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1029.487574] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 23:17:50 (1787368670) [ 1029.595354] LNet: 52443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.526698] Key type .llcrypt unregistered [ 1030.533898] Key type ._llcrypt unregistered [ 1037.891800] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1038.088294] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1038.100614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1038.123592] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1040.927399] Key type ._llcrypt registered [ 1040.935426] Key type .llcrypt registered [ 1041.258769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.293802] alg: No test for adler32 (adler32-zlib) [ 1042.323836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.471268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.505027] LNet: Added LNI 192.168.202.59@tcp [8/512/0/180] [ 1042.515815] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1042.519237] LNet: 52776:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.645767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.666756] LNet: 52795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.677475] LNetError: 52795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.684439] LNet: Removed LNI 10.1.2.3@tcp [ 1042.689482] LNetError: 52795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.711141] LNet: Removed LNI 192.168.202.59@tcp [ 1042.747369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.888356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.953532] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1043.107358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.147641] LNet: 52819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.161401] LNetError: 52819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.177268] LNet: Removed LNI 192.168.202.59@tcp [ 1043.195149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.263388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.296750] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1043.304892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1043.308780] LNet: 52827:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.396919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.427649] LNet: 52845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.436494] LNetError: 52845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.447742] LNet: Removed LNI 10.1.2.3@tcp [ 1043.452247] LNetError: 52845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.465706] LNet: Removed LNI 192.168.202.59@tcp [ 1043.482524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.576227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.600183] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1043.604421] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1043.608199] LNet: 52853:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.859455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.908243] LNet: 52879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.916434] LNetError: 52879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.919495] LNet: Removed LNI 10.1.2.3@tcp [ 1043.923712] LNetError: 52879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.965884] LNet: Removed LNI 192.168.202.59@tcp [ 1044.033160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.097524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1044.113517] LNet: Added LNI 192.168.202.59@tcp [8/256/0/182] [ 1044.118981] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1044.201597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1044.226535] LNet: 52907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.230505] LNetError: 52907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1044.235260] LNet: Removed LNI 10.1.2.3@tcp [ 1044.239755] LNetError: 52907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.252386] LNet: Removed LNI 192.168.202.59@tcp [ 1044.288848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.396483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1044.437602] LNet: Added LNI 192.168.202.59@tcp [8/256/0/182] [ 1044.440525] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1044.445592] LNet: 52915:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1044.532924] LNet: Removed LNI 10.1.2.3@tcp [ 1044.542443] LNet: Removed LNI 192.168.202.59@tcp [ 1045.435138] Key type .llcrypt unregistered [ 1045.439519] Key type ._llcrypt unregistered [ 1058.091364] Key type ._llcrypt registered [ 1058.094247] Key type .llcrypt registered [ 1059.185282] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 23:18:20 (1787368700) [ 1061.062831] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1063.121266] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 23:18:23 (1787368703) [ 1063.637783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1063.657893] alg: No test for adler32 (adler32-zlib) [ 1064.593590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.681723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1064.707919] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1065.935382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.59 [ 1065.962911] LNet: 54185:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1065.971807] LNet: Removed LNI 192.168.202.59@tcp [ 1073.494342] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 23:18:33 (1787368713) [ 1073.715820] LNet: 54746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.722452] LNetError: 54746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1073.757258] LNet: Removed LNI 192.168.202.59@tcp [ 1074.537499] Key type .llcrypt unregistered [ 1074.541517] Key type ._llcrypt unregistered [ 1081.554349] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1081.857815] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1084.720404] Key type ._llcrypt registered [ 1084.722739] Key type .llcrypt registered [ 1084.937669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1084.954927] alg: No test for adler32 (adler32-zlib) [ 1086.455795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1093.449324] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 23:18:54 (1787368734) [ 1093.656191] LNet: 55673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.666256] LNetError: 55673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1093.680400] LNet: Removed LNI 10.1.2.3@tcp [ 1094.259152] Key type .llcrypt unregistered [ 1094.260829] Key type ._llcrypt unregistered [ 1100.338147] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1100.632458] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1100.637384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1100.647772] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1103.589663] Key type ._llcrypt registered [ 1103.591850] Key type .llcrypt registered [ 1103.758468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1103.790784] alg: No test for adler32 (adler32-zlib) [ 1104.790730] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1112.144597] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 23:19:13 (1787368753) [ 1114.212802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1115.955707] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 23:19:17 (1787368757) [ 1116.457437] LNet: 56862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.468983] LNetError: 56862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1116.487330] LNet: Removed LNI 10.1.2.3@tcp [ 1117.409837] Key type .llcrypt unregistered [ 1117.412464] Key type ._llcrypt unregistered [ 1125.893359] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1126.207584] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1126.213265] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1126.216342] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1129.012906] Key type ._llcrypt registered [ 1129.017429] Key type .llcrypt registered [ 1129.227447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1129.245761] alg: No test for adler32 (adler32-zlib) [ 1130.216720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1130.276976] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1139.718211] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 23:19:40 (1787368780) [ 1140.331824] LNet: 58096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.349070] LNetError: 58096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1140.371615] LNet: Removed LNI 10.1.2.3@tcp [ 1141.287589] Key type .llcrypt unregistered [ 1141.289950] Key type ._llcrypt unregistered [ 1147.933814] Key type ._llcrypt registered [ 1147.936501] Key type .llcrypt registered [ 1148.223563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1148.257264] alg: No test for adler32 (adler32-zlib) [ 1149.152407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1149.175819] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1157.603324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 1158.538256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_interrupt -r 1 [ 1158.562755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_interrupt -r 1 [ 1158.594520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1158.609995] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1158.821421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.835218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.848748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.862259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.879107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.904095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.934401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.961923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.978562] LNet: 58539:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.995228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.024473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.056229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.089705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.138642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.174221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.213822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1159.871172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_dropped -r 1 [ 1159.911413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_dropped -r 1 [ 1159.953365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1159.974361] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1160.296475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.329632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.361026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.406726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.451920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.484960] LNet: 1 local NIs in recovery (showing 1): 192.168.202.59@tcp [ 1160.522467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.554575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.583404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.602736] LNet: 58648:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.626485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.665673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.721985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.769462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.813153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.860988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.915169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1161.731416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_aborted -r 1 [ 1161.794368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_aborted -r 1 [ 1161.834377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1161.857628] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1161.861897] LNet: Skipped 1 previous similar message [ 1162.063250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.086445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1162.124112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.148495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.173177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.203627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.232268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.255533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.270848] LNet: 58758:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.292806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.318331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.359349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.389231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.411777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.435112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.459717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1162.899219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_no_route -r 1 [ 1162.928479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_no_route -r 1 [ 1162.976167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1163.330500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.380538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.427160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.469523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.529747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.583520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.660494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.698388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.744861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.782164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.836868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.878829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.904281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.956825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.006300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1164.528393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_timeout -r 1 [ 1164.579433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_timeout -r 1 [ 1164.666334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1164.723068] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1164.730952] LNet: Skipped 1 previous similar message [ 1165.015610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.056506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.102976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.124576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.180480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.234925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.304512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.346964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.369304] LNet: 58976:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.384031] LNet: 58976:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1165.422699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.479070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.550663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.623864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.673462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.753212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.810696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1166.459462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_error -r 1 [ 1166.514405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_error -r 1 [ 1166.559401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1166.862323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.921490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.969527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.010816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.043858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.081348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.118458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.156427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.188797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.215154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.244762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.279140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.307156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.337617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.372346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.740389] LNet: 59131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.746183] LNetError: 59131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.820974] LNet: Removed LNI 192.168.202.59@tcp [ 1169.244650] Key type .llcrypt unregistered [ 1169.248318] Key type ._llcrypt unregistered [ 1188.633449] Key type ._llcrypt registered [ 1188.643578] Key type .llcrypt registered [ 1190.277644] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 23:20:30 (1787368830) [ 1191.544492] Key type .llcrypt unregistered [ 1191.547588] Key type ._llcrypt unregistered [ 1200.985598] Key type ._llcrypt registered [ 1200.992528] Key type .llcrypt registered [ 1201.451274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1201.496643] alg: No test for adler32 (adler32-zlib) [ 1202.486646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1202.520756] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1213.207900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 1224.494622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1224.543153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1224.585407] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1226.561515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_interrupt -r 1 [ 1226.593424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_interrupt -r 1 [ 1226.617683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_interrupt -r 1 [ 1226.645753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_interrupt -r 1 [ 1226.672549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_interrupt -r 1 [ 1226.697824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_interrupt -r 1 [ 1226.747660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_interrupt -r 1 [ 1226.801900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_interrupt -r 1 [ 1226.860954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1226.878069] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1226.887464] LNet: 1 local NIs in recovery (showing 1): 192.168.202.59@tcp [ 1227.137130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.195616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1227.222652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.241487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.261718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.282351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.335550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.394306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.433067] LNet: 60606:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.476653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.525715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.560931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.590869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.637366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.671731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.700337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1228.227791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_dropped -r 1 [ 1228.264355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_dropped -r 1 [ 1228.293456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_dropped -r 1 [ 1228.328527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_dropped -r 1 [ 1228.362713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_dropped -r 1 [ 1228.405907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_dropped -r 1 [ 1228.440728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_dropped -r 1 [ 1228.490500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_dropped -r 1 [ 1228.524443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1228.550569] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1228.554354] LNet: Skipped 2 previous similar messages [ 1228.771980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.801174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.834604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.862816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.892464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.942750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.001388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.035635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.051165] LNet: 60728:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.068372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.108799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.150991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.203344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.247174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.280579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.339545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1229.781561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_aborted -r 1 [ 1229.811931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_aborted -r 1 [ 1229.843165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_aborted -r 1 [ 1229.876790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_aborted -r 1 [ 1229.903328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_aborted -r 1 [ 1229.935463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_aborted -r 1 [ 1229.964227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_aborted -r 1 [ 1229.988766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_aborted -r 1 [ 1230.018451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1230.032287] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1230.035985] LNet: Skipped 2 previous similar messages [ 1230.303616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.329174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.355875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.388900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.424142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.474763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.544762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.588522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.613899] LNet: 60849:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.638759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.691445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.740078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.799822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.842864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.920750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.016129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1231.989244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_no_route -r 1 [ 1232.054567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_no_route -r 1 [ 1232.107133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_no_route -r 1 [ 1232.160207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_no_route -r 1 [ 1232.227822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_no_route -r 1 [ 1232.264501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_no_route -r 1 [ 1232.307515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_no_route -r 1 [ 1232.382783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_no_route -r 1 [ 1232.469389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1232.525319] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1232.533398] LNet: Skipped 2 previous similar messages [ 1232.849656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.886902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.927690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.975462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1233.018529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1233.040908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1233.075773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1233.100730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1233.123585] LNet: 60970:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1233.147731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1233.197621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1233.229436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1233.260553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1233.290974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.321764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.349479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1233.781396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_timeout -r 1 [ 1233.813752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_timeout -r 1 [ 1233.841596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_timeout -r 1 [ 1233.882984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_timeout -r 1 [ 1233.932904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_timeout -r 1 [ 1233.975244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_timeout -r 1 [ 1234.014471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_timeout -r 1 [ 1234.058912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_timeout -r 1 [ 1234.104533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1234.449769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.492674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1234.530972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.596609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.646796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.684807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.726854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.768700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.848634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.922255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.998761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1235.053521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.095466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.179118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.226522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1235.951641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_error -r 1 [ 1236.007085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_error -r 1 [ 1236.070083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_error -r 1 [ 1236.143490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_error -r 1 [ 1236.234285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_error -r 1 [ 1236.297960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_error -r 1 [ 1236.338455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_error -r 1 [ 1236.388699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_error -r 1 [ 1236.437803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1236.722475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.771274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.825683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.870722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.931665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.974429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.010669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.038417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.073853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.113760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.140806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.169413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.196804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.229494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.262344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1242.768245] LNet: 61293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.776981] LNetError: 61293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1242.790190] LNet: Removed LNI 192.168.202.59@tcp [ 1243.822461] LNet: Removed LNI 192.168.202.59@tcp1 [ 1244.369861] Key type .llcrypt unregistered [ 1244.371970] Key type ._llcrypt unregistered [ 1262.066428] Key type ._llcrypt registered [ 1262.068847] Key type .llcrypt registered [ 1263.167304] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 23:21:44 (1787368904) [ 1264.352620] Key type .llcrypt unregistered [ 1264.354208] Key type ._llcrypt unregistered [ 1271.270877] Key type ._llcrypt registered [ 1271.272148] Key type .llcrypt registered [ 1271.427956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1271.441683] alg: No test for adler32 (adler32-zlib) [ 1272.343882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1272.372157] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1278.980930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 1279.664713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e remote_dropped -r 1 [ 1279.705430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e remote_dropped -r 1 [ 1279.730801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1279.743119] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1279.901519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.923899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.950766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.981287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.003721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.031693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.057071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.085897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.100989] LNet: 62549:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.126362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.166820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.203242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.233901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.314855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.355327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.402739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1281.057145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e remote_error -r 1 [ 1281.116720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e remote_error -r 1 [ 1281.168361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1281.190380] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1281.376618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1281.422680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1281.479418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1281.522624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1281.565978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1281.594137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1281.619287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1281.651752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1281.668714] LNet: 62658:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1281.689582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1281.717883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1281.745980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1281.776949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1281.804991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.832870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1281.865599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1282.323602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e remote_timeout -r 1 [ 1282.357891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e remote_timeout -r 1 [ 1282.398629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1282.418138] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1282.696843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1282.727664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1282.746964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1282.771910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1282.815752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1282.836733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1282.879980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1282.938573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1282.966168] LNet: 62767:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1283.005400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.058813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.103823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.145264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.194273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.252364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.286675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1283.659522] LNet: 62812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.669798] LNetError: 62812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.680821] LNet: Removed LNI 192.168.202.59@tcp [ 1284.052408] Key type .llcrypt unregistered [ 1284.054694] Key type ._llcrypt unregistered [ 1302.035690] Key type ._llcrypt registered [ 1302.037880] Key type .llcrypt registered [ 1303.238096] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 23:22:24 (1787368944) [ 1304.544771] Key type .llcrypt unregistered [ 1304.547995] Key type ._llcrypt unregistered [ 1311.816108] Key type ._llcrypt registered [ 1311.818879] Key type .llcrypt registered [ 1312.102775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1312.151447] alg: No test for adler32 (adler32-zlib) [ 1313.226987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1313.259353] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1321.386619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 1331.787324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1331.876634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1331.917295] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1333.610905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e remote_dropped -r 1 [ 1333.632188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e remote_dropped -r 1 [ 1333.654735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e remote_dropped -r 1 [ 1333.678276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e remote_dropped -r 1 [ 1333.700199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e remote_dropped -r 1 [ 1333.725192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e remote_dropped -r 1 [ 1333.756591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e remote_dropped -r 1 [ 1333.782632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e remote_dropped -r 1 [ 1333.811811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1333.825448] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1333.829462] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.159@tcp [ 1333.973265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.995822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1334.015947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1334.045932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1334.079071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1334.096555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1334.113843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1334.134511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1334.156650] LNet: 64284:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1334.175184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1334.213766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1334.246173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1334.283579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1334.315259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.335648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1334.366130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1334.403145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.761458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e remote_error -r 1 [ 1334.777672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e remote_error -r 1 [ 1334.793912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e remote_error -r 1 [ 1334.810393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e remote_error -r 1 [ 1334.832931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e remote_error -r 1 [ 1334.850303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e remote_error -r 1 [ 1334.870508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e remote_error -r 1 [ 1334.895504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e remote_error -r 1 [ 1334.924240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1334.936142] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1334.940223] LNet: Skipped 2 previous similar messages [ 1335.088847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1335.111576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1335.140757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.166691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1335.196261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1335.227644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1335.259948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1335.283993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1335.297449] LNet: 64407:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1335.309787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1335.335774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1335.364113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1335.414675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1335.450236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1335.492763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1335.579908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1335.643322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.262689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e remote_timeout -r 1 [ 1336.313151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e remote_timeout -r 1 [ 1336.362487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e remote_timeout -r 1 [ 1336.414528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e remote_timeout -r 1 [ 1336.454638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e remote_timeout -r 1 [ 1336.497783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e remote_timeout -r 1 [ 1336.545268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e remote_timeout -r 1 [ 1336.594673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e remote_timeout -r 1 [ 1336.634605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.159@tcp [ 1336.656223] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1336.909472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1336.929847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1336.946663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1336.963557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1336.978027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1336.996878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1337.019969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1337.039196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1337.051554] LNet: 64531:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1337.067855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1337.086463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1337.109857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1337.136626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1337.180051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1337.219077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1337.243747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1337.270864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.887649] LNet: 64613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.898210] LNetError: 64613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.967260] LNet: Removed LNI 192.168.202.59@tcp [ 1343.975109] LNetError: 64613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.998692] LNet: Removed LNI 192.168.202.59@tcp1 [ 1344.528723] Key type .llcrypt unregistered [ 1344.535705] Key type ._llcrypt unregistered [ 1363.927155] Key type ._llcrypt registered [ 1363.929685] Key type .llcrypt registered [ 1365.374961] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 23:23:26 (1787369006) [ 1367.015033] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1368.745697] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 23:23:29 (1787369009) [ 1369.693490] Key type .llcrypt unregistered [ 1369.698135] Key type ._llcrypt unregistered [ 1377.410625] Key type ._llcrypt registered [ 1377.412469] Key type .llcrypt registered [ 1377.581369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1377.591766] alg: No test for adler32 (adler32-zlib) [ 1378.517121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1378.545275] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1386.069051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 1386.775735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e network_timeout -r 1 [ 1386.867470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e network_timeout -r 1 [ 1386.918852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 1386.948591] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1387.208380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1387.248580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1387.284429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1387.312441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1387.343908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1387.385230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1387.418964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1387.489302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1387.517334] LNet: 66166:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1387.549458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1387.587381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1387.623450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1387.675287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1387.764922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1387.806611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1388.202913] LNet: 66209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.209693] LNetError: 66209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.242481] LNet: Removed LNI 192.168.202.59@tcp [ 1388.658749] Key type .llcrypt unregistered [ 1388.663538] Key type ._llcrypt unregistered [ 1407.601919] Key type ._llcrypt registered [ 1407.603206] Key type .llcrypt registered [ 1407.714823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1407.733067] alg: No test for adler32 (adler32-zlib) [ 1408.564886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1408.584388] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1416.574167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 1426.476918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.533406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1426.570367] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1428.100320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e network_timeout -r 1 [ 1428.146689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e network_timeout -r 1 [ 1428.191456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e network_timeout -r 1 [ 1428.256319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e network_timeout -r 1 [ 1428.341313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e network_timeout -r 1 [ 1428.396902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e network_timeout -r 1 [ 1428.456970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e network_timeout -r 1 [ 1428.502817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e network_timeout -r 1 [ 1428.552595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 1428.567916] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1428.811858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1428.844904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1428.878984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1428.909844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1428.937634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1428.961946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1428.989670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1429.020968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1429.034270] LNet: 67094:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1429.057501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1429.087635] LNet: 1 local NIs in recovery (showing 1): 192.168.202.59@tcp [ 1429.092247] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1429.096150] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.159@tcp [ 1429.113662] LNet: Skipped 1 previous similar message [ 1429.140903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1429.202173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1429.239185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1429.271604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1429.300849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1433.994537] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1434.606405] LNet: 67172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.613357] LNetError: 67172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.627353] LNet: Removed LNI 192.168.202.59@tcp [ 1434.642740] LNet: Removed LNI 192.168.202.59@tcp1 [ 1435.253640] Key type .llcrypt unregistered [ 1435.256738] Key type ._llcrypt unregistered [ 1455.633696] Key type ._llcrypt registered [ 1455.636351] Key type .llcrypt registered [ 1457.229564] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 23:24:57 (1787369097) [ 1457.946547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1457.981821] alg: No test for adler32 (adler32-zlib) [ 1458.970735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.042350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1459.060383] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1459.882412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1459.921095] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1460.944329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 1461.005191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1461.041806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1461.064350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1461.087889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1461.127728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1461.202232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.59@tcp [ 1461.231375] LNetError: 68276:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.59@tcp to recovery [ 1462.047408] LNet: 1 local NIs in recovery (showing 1): 192.168.202.59@tcp [ 1462.047502] LNet: There was an unexpected network error while writing to 192.168.202.59: rc = -22 [ 1464.096929] LNet: There was an unexpected network error while writing to 192.168.202.59: rc = -22 [ 1465.003771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1468.191693] LNet: There was an unexpected network error while writing to 192.168.202.59: rc = -22 [ 1468.684526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1468.736891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1468.803128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1468.834963] LNet: 68383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.844612] LNetError: 68383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1468.852250] LNet: Removed LNI 192.168.202.59@tcp [ 1468.871903] LNet: Removed LNI 192.168.202.59@tcp1 [ 1468.889808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.935482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1468.959798] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1469.841346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1469.880334] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1470.993330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 1471.042557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1471.093441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1471.196867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1471.284239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1471.342911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1471.390887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1471.429031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.59@tcp [ 1471.462816] LNetError: 68656:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.59@tcp to recovery [ 1471.967617] LNet: There was an unexpected network error while writing to 192.168.202.59: rc = -22 [ 1475.092964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1478.123225] LNet: There was an unexpected network error while writing to 192.168.202.59: rc = -22 [ 1478.134245] LNet: Skipped 1 previous similar message [ 1483.256544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1483.310693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1483.340736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1490.878694] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 23:25:31 (1787369131) [ 1490.977697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1491.012476] LNet: 69366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.028312] LNetError: 69366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.037065] LNetError: 69366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1491.050600] LNet: Removed LNI 192.168.202.59@tcp [ 1491.081228] LNet: Removed LNI 192.168.202.59@tcp1 [ 1491.124297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.243443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1491.292418] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1492.408403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1492.450570] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1493.414071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 1493.515423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1493.564626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1493.595277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1493.671575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1493.731504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1493.778440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.59@tcp --health 0 [ 1493.932065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1494.305976] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.59@tcp [ 1494.306119] LNet: There was an unexpected network error while writing to 192.168.202.59: rc = -22 [ 1494.320717] LNet: Skipped 1 previous similar message [ 1509.771422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1509.922579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1509.947939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1509.996971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.59@tcp --health 500 [ 1510.129356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1516.981285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1517.055644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.091970] LNet: 69866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.110678] LNetError: 69866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.122057] LNetError: 69866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1517.133836] LNet: Removed LNI 192.168.202.59@tcp [ 1517.155910] LNet: Removed LNI 192.168.202.59@tcp1 [ 1517.191714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.263205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.295767] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1518.065771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.094793] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1519.152683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 1519.184409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.276787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1519.331330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1519.371693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1519.432774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1519.470761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1519.508205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.59@tcp --health 0 [ 1520.288212] LNet: There was an unexpected network error while writing to 192.168.202.59: rc = -22 [ 1520.292477] LNet: Skipped 3 previous similar messages [ 1520.913861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1531.395187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1531.432436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1531.473996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1538.631349] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 23:26:19 (1787369179) [ 1539.358507] LNet: 70913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.366258] LNetError: 70913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.372551] LNetError: 70913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1539.382481] LNet: Removed LNI 192.168.202.59@tcp [ 1539.408500] LNet: Removed LNI 192.168.202.59@tcp1 [ 1540.346689] Key type .llcrypt unregistered [ 1540.350086] Key type ._llcrypt unregistered [ 1546.504899] Key type ._llcrypt registered [ 1546.509780] Key type .llcrypt registered [ 1546.773279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1546.802068] alg: No test for adler32 (adler32-zlib) [ 1547.795712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1547.834225] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1556.724875] Lustre: DEBUG MARKER: Initial discovery [ 1556.923522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.159@tcp [ 1559.220415] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1559.374184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1559.440522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.159@tcp [ 1559.483474] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1559.505396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1559.543471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.159@tcp -r 1 -m GET -l 3 [ 1559.570675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.159@tcp -r 1 -m GET -e local_error [ 1559.614555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.159@tcp -r 1 -m PUT -l 6 [ 1560.162899] LNet: 1 local NIs in recovery (showing 1): 192.168.202.59@tcp [ 1560.918243] Lustre: DEBUG MARKER: Force oleg259-client.virtnet to discover 192.168.202.159@tcp (in background) [ 1561.210798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.159@tcp [ 1562.423588] Lustre: DEBUG MARKER: Force oleg259-server to discover 192.168.202.59@tcp [ 1564.322750] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1564.965705] Lustre: DEBUG MARKER: Wait for 71680 [ 1566.475304] Lustre: DEBUG MARKER: Finished wait on 71680 [ 1566.538662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1566.623780] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1566.630115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1567.002085] LNet: 72134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.006882] LNetError: 72134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.031764] LNet: Removed LNI 192.168.202.59@tcp [ 1567.524549] Key type .llcrypt unregistered [ 1567.526570] Key type ._llcrypt unregistered [ 1583.640633] Key type ._llcrypt registered [ 1583.644034] Key type .llcrypt registered [ 1584.991959] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 23:27:05 (1787369225) [ 1585.579486] Key type .llcrypt unregistered [ 1585.582609] Key type ._llcrypt unregistered [ 1591.550612] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1591.780828] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1591.791173] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1591.804456] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1594.416777] Key type ._llcrypt registered [ 1594.418472] Key type .llcrypt registered [ 1594.553751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1594.574471] alg: No test for adler32 (adler32-zlib) [ 1595.575814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.635471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1595.671768] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1596.358244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1596.376906] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1605.186975] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 23:27:26 (1787369246) [ 1605.384587] LNet: 74208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.396129] LNetError: 74208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1605.414194] LNet: Removed LNI 10.1.2.3@tcp [ 1605.428393] LNet: Removed LNI 192.168.202.59@tcp [ 1606.105526] Key type .llcrypt unregistered [ 1606.114060] Key type ._llcrypt unregistered [ 1612.876437] Key type ._llcrypt registered [ 1612.878880] Key type .llcrypt registered [ 1613.037726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1613.048565] alg: No test for adler32 (adler32-zlib) [ 1613.976750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.074608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.113818] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1614.987805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1614.997668] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1615.844401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.59@tcp [ 1616.927345] LNetError: Unexpected error -22 connecting to 192.168.202.59@tcp at host 192.168.202.59:988 [ 1621.811749] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 23:27:43 (1787369263) [ 1622.047981] LNet: 75347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.063887] LNetError: 75347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1622.073675] LNet: Removed LNI 10.1.2.3@tcp [ 1622.085771] LNet: Removed LNI 192.168.202.59@tcp [ 1622.731671] Key type .llcrypt unregistered [ 1622.733728] Key type ._llcrypt unregistered [ 1629.405561] Key type ._llcrypt registered [ 1629.408675] Key type .llcrypt registered [ 1629.674049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1629.691193] alg: No test for adler32 (adler32-zlib) [ 1630.616619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.718350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1630.773816] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1631.674427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1631.700404] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 1632.794819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.59@tcp1 --nid 192.168.202.59@tcp2 [ 1636.564425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.59@tcp1 192.168.202.59@tcp2 [ 1636.584188] LNetError: 76187:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.59@tcp1 to 192.168.202.59@tcp2 - Route table may be misconfigured [ 1636.603944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.59@tcp2 192.168.202.59@tcp1 [ 1643.413598] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 23:28:04 (1787369284) [ 1643.663924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.695496] LNet: 76761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.706731] LNetError: 76761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.717750] LNet: Removed LNI 192.168.202.59@tcp1 [ 1644.779928] LNet: Removed LNI 192.168.202.59@tcp2 [ 1644.853239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.909968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.935395] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1645.689249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1645.717023] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1646.414267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 1646.450545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.59@tcp 192.168.202.59@tcp [ 1646.482900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.59@tcp1 192.168.202.59@tcp1 [ 1646.514773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.59@tcp -d 192.168.202.59@tcp -e network_timeout [ 1646.550825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -e network_timeout [ 1646.586299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -e network_timeout [ 1646.625758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -e network_timeout [ 1646.660233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1646.686053] LNet: There was an unexpected network error while writing to 192.168.202.59: rc = -22 [ 1646.715481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1646.782786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1646.872316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1646.949129] LNet: 1 local NIs in recovery (showing 1): 192.168.202.59@tcp [ 1653.670599] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 23:28:14 (1787369294) [ 1654.024303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.057366] LNet: 77622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.068533] LNetError: 77622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.072273] LNetError: 77622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1654.085733] LNet: Removed LNI 192.168.202.59@tcp [ 1655.171645] LNet: Removed LNI 192.168.202.59@tcp1 [ 1655.213806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.357785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1655.692293] LNet: 77664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.092395] Key type .llcrypt unregistered [ 1656.094679] Key type ._llcrypt unregistered [ 1668.747267] Key type ._llcrypt registered [ 1668.756309] Key type .llcrypt registered [ 1670.375626] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 23:28:30 (1787369310) [ 1670.778627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1670.814153] alg: No test for adler32 (adler32-zlib) [ 1671.741389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1673.196353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1675.034631] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 23:28:36 (1787369316) [ 1675.507280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.532313] LNet: 78765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.552580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.602158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1675.622371] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1676.266304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1676.290725] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1677.169759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1677.209938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp1 [ 1677.241655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp1 [ 1683.769543] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 23:28:44 (1787369324) [ 1686.215138] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg259-server [ 1688.128236] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 23:28:49 (1787369329) [ 1690.117295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg259-server [ 1692.014320] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 23:28:53 (1787369333) [ 1694.699644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg259-server [ 1696.077606] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 23:28:57 (1787369337) [ 1699.412363] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg259-server [ 1701.025408] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 23:29:02 (1787369342) [ 1704.433032] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg259-server [ 1707.824448] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 23:29:07 (1787369347) [ 1710.638054] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg259-server [ 1713.936576] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 23:29:13 (1787369353) [ 1717.673921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg259-server [ 1719.996059] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 23:29:21 (1787369361) [ 1722.503424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg259-server [ 1725.426919] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 23:29:25 (1787369365) [ 1727.846176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg259-server [ 1729.951871] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 23:29:30 (1787369370) [ 1730.118176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.181844] LNet: 82639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.192461] LNetError: 82639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.231183] LNet: Removed LNI 192.168.202.59@tcp [ 1731.235234] LNetError: 82639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.265787] LNet: Removed LNI 192.168.202.59@tcp1 [ 1731.339489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.445483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1731.474550] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1732.367435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1732.476583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1732.690987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.732837] LNet: 82793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.740044] LNetError: 82793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.802875] LNet: Removed LNI 192.168.202.59@tcp [ 1733.852314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.950431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1733.982993] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1735.040851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1735.310207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.344444] LNet: 82938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.359198] LNetError: 82938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.377120] LNet: Removed LNI 192.168.202.59@tcp [ 1735.397621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.527241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1735.585825] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1736.723079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1736.841231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1737.084385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.152827] LNet: Removed LNI 192.168.202.59@tcp [ 1738.210319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.278250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1738.337363] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1739.126298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1739.384373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.432553] LNet: 83231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.436707] LNet: 83231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1739.444987] LNetError: 83231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.457758] LNetError: 83231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1739.489217] LNet: Removed LNI 192.168.202.59@tcp [ 1739.560512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.626640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.657397] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1740.443954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1740.544115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1740.754786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.806309] LNet: Removed LNI 192.168.202.59@tcp [ 1741.848097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.986364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1742.047375] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1743.000815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1744.611747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.702884] LNet: Removed LNI 192.168.202.59@tcp [ 1745.761817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.865480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1745.893982] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1746.728332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1746.829453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1747.055375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.136669] LNet: Removed LNI 192.168.202.59@tcp [ 1748.175092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.230727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1748.250626] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1749.195907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1749.483475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.512792] LNet: 83825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.518899] LNet: 83825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1749.532207] LNetError: 83825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.540598] LNetError: 83825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1750.569529] LNet: Removed LNI 192.168.202.59@tcp [ 1750.587422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.629751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.673612] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1751.398223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1751.460561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1751.692958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.742957] LNet: Removed LNI 192.168.202.59@tcp [ 1752.773732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.864799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1752.936042] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1753.703522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1753.943121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.055572] LNet: Removed LNI 192.168.202.59@tcp [ 1755.114927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.183430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.253606] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1756.017771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1756.114395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1757.549173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.635559] LNet: Removed LNI 192.168.202.59@tcp [ 1758.671422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.770134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1758.805621] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1759.410359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 1759.589500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.680696] LNet: Removed LNI 192.168.202.59@tcp [ 1760.729368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.800959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1760.827885] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1761.607172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1761.656760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.682109] LNet: Removed LNI 192.168.202.59@tcp [ 1761.704734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.744799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.766442] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1762.425441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1762.522714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.559394] LNet: Removed LNI 192.168.202.59@tcp [ 1762.581988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.650468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1762.738279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.766210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.801521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1762.853343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.892505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.974402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1763.066744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.102781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.153662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1763.177460] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1763.805661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.835235] LNet: Removed LNI 192.168.202.59@tcp [ 1763.855507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.960383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1771.419987] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 23:30:12 (1787369412) [ 1771.803664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.840664] LNet: 85441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.855446] LNet: 85441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1771.891809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.995645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1772.038743] LNet: Added LNI 192.168.202.59@tcp231 [8/256/0/180] [ 1773.145751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.210661] LNetError: 85579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.218863] LNetError: 85579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1773.236767] LNet: Removed LNI 192.168.202.59@tcp231 [ 1773.275653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.382340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1773.440529] LNet: Added LNI 192.168.202.59@tcp231 [8/256/0/180] [ 1773.448369] LNet: 85587:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1774.038335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1774.074894] LNet: Removed LNI 192.168.202.59@tcp231 [ 1774.096188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1774.141389] LNet: Added LNI 192.168.202.59@tcp231 [8/256/0/180] [ 1775.265742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.298231] LNet: Removed LNI 192.168.202.59@tcp231 [ 1775.318544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.427140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1775.458791] LNet: Added LNI 192.168.202.59@tcp231 [8/256/0/0] [ 1775.465478] LNet: 85740:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1776.133968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1776.158848] LNet: Removed LNI 192.168.202.59@tcp231 [ 1776.176036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1776.212127] LNet: Added LNI 192.168.202.59@tcp231 [8/256/0/0] [ 1777.707913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.748270] LNet: Removed LNI 192.168.202.59@tcp231 [ 1777.779233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.903833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1777.945611] LNet: Added LNI 192.168.202.59@tcp231 [8/256/0/60] [ 1777.951518] LNet: 85891:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1778.878769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1778.929844] LNet: Removed LNI 192.168.202.59@tcp231 [ 1778.953520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1779.018899] LNet: Added LNI 192.168.202.59@tcp231 [8/256/0/60] [ 1780.910370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.968601] LNet: Removed LNI 192.168.202.59@tcp231 [ 1780.999319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.076659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1781.115389] LNet: Added LNI 192.168.202.59@tcp231 [8/256/0/180] [ 1781.120841] LNet: 86042:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1781.749468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1781.811142] LNet: Removed LNI 192.168.202.59@tcp231 [ 1781.852872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1781.894795] LNet: Added LNI 192.168.202.59@tcp231 [8/256/0/180] [ 1790.830586] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 23:30:31 (1787369431) [ 1791.321576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.402207] LNet: Removed LNI 192.168.202.59@tcp231 [ 1791.447834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.591317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1791.644138] LNet: Added LNI 192.168.202.59@tcp232 [8/256/0/180] [ 1792.649991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1792.666547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1792.685251] LNet: Removed LNI 192.168.202.59@tcp232 [ 1792.696446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1792.714350] LNet: Added LNI 192.168.202.59@tcp232 [8/256/0/180] [ 1793.288572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1793.329853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1793.367311] LNet: Removed LNI 192.168.202.59@tcp232 [ 1793.383661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1793.407597] LNet: Added LNI 192.168.202.59@tcp232 [8/256/0/180] [ 1794.307337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1794.337947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1794.370417] LNet: Removed LNI 192.168.202.59@tcp232 [ 1794.390727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1794.437150] LNet: Added LNI 192.168.202.59@tcp232 [8/256/0/180] [ 1795.375618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1795.419536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1795.450565] LNet: Removed LNI 192.168.202.59@tcp232 [ 1795.469313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1801.891109] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 23:30:43 (1787369443) [ 1803.188640] Key type .llcrypt unregistered [ 1803.191271] Key type ._llcrypt unregistered [ 1811.594242] Key type ._llcrypt registered [ 1811.595954] Key type .llcrypt registered [ 1811.812552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1811.824842] alg: No test for adler32 (adler32-zlib) [ 1812.744437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1812.782888] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1819.775447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 1830.272289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.345675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1830.405710] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1832.363551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_interrupt -r 2 [ 1832.406344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_interrupt -r 2 [ 1832.458982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_interrupt -r 2 [ 1832.493183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_interrupt -r 2 [ 1832.529656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_interrupt -r 2 [ 1832.569560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_interrupt -r 2 [ 1832.642178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_interrupt -r 2 [ 1832.711339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_interrupt -r 2 [ 1832.793224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.59@tcp 192.168.202.159@tcp [ 1832.853725] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1832.868151] LNet: 1 local NIs in recovery (showing 1): 192.168.202.59@tcp [ 1833.403609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1833.446502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1833.491996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1833.527636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1833.570491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1833.617692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1833.656649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1833.729632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1833.767234] LNet: 88547:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1833.795608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1833.882707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1833.935974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1834.011694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1834.116201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1834.185335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1835.057203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_dropped -r 2 [ 1835.081521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_dropped -r 2 [ 1835.116953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_dropped -r 2 [ 1835.154886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_dropped -r 2 [ 1835.201375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_dropped -r 2 [ 1835.247603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_dropped -r 2 [ 1835.315790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_dropped -r 2 [ 1835.407168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_dropped -r 2 [ 1835.533511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.59@tcp 192.168.202.159@tcp [ 1835.558109] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1835.956703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1835.990812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1836.022580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1836.079838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1836.129496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1836.175848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1836.238105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1836.282351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1836.315346] LNet: 88667:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1836.334276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1836.420218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1836.457935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1836.524774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1836.591834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1836.648204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1837.132420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_aborted -r 2 [ 1837.200125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_aborted -r 2 [ 1837.269954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_aborted -r 2 [ 1837.300219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_aborted -r 2 [ 1837.331738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_aborted -r 2 [ 1837.363394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_aborted -r 2 [ 1837.413144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_aborted -r 2 [ 1837.524345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_aborted -r 2 [ 1837.591167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.59@tcp 192.168.202.159@tcp [ 1837.609856] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1837.810627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1837.840289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1837.886437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1837.920700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1837.947879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1837.979619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1838.004203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1838.032990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1838.044320] LNet: 88788:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1838.059301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1838.083186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1838.107364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1838.135279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1838.157475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1838.183067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1838.688732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_no_route -r 2 [ 1838.736059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_no_route -r 2 [ 1838.803114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_no_route -r 2 [ 1838.862778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_no_route -r 2 [ 1838.914243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_no_route -r 2 [ 1838.974304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_no_route -r 2 [ 1839.031247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_no_route -r 2 [ 1839.083618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_no_route -r 2 [ 1839.158771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.59@tcp 192.168.202.159@tcp [ 1839.522962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1839.546546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1839.569673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1839.638894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1839.701302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1839.744196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1839.781455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1839.817643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1839.842973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1839.887628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1839.939236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1839.972190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1840.014878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1840.048264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1840.566928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e local_timeout -r 2 [ 1840.603747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e local_timeout -r 2 [ 1840.636651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e local_timeout -r 2 [ 1840.665810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e local_timeout -r 2 [ 1840.691190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e local_timeout -r 2 [ 1840.724340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e local_timeout -r 2 [ 1840.762588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e local_timeout -r 2 [ 1840.812441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e local_timeout -r 2 [ 1840.866991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.59@tcp 192.168.202.159@tcp [ 1840.897674] LNet: There was an unexpected network error while writing to 192.168.202.159: rc = -22 [ 1840.910954] LNet: Skipped 1 previous similar message [ 1841.257839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1841.306742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1841.366224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1841.413691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1841.480916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1841.516477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1841.574853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1841.626464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1841.663201] LNet: 89028:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1841.675460] LNet: 89028:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1841.718512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1841.795509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1841.850718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1841.897404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1841.959773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1841.998232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1842.852516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp -m GET -e remote_dropped -r 2 [ 1842.916230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.159@tcp1 -m GET -e remote_dropped -r 2 [ 1842.982424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp -m GET -e remote_dropped -r 2 [ 1843.036043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp -d 192.168.202.59@tcp1 -m GET -e remote_dropped -r 2 [ 1843.126347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp -m GET -e remote_dropped -r 2 [ 1843.210196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.59@tcp1 -d 192.168.202.159@tcp1 -m GET -e remote_dropped -r 2 [ 1843.327843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp -m GET -e remote_dropped -r 2 [ 1843.387506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.59@tcp1 -d 192.168.202.59@tcp1 -m GET -e remote_dropped -r 2 [ 1843.471953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.59@tcp 192.168.202.159@tcp [ 1843.519024] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.159@tcp [ 1843.815859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1843.844133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1843.866862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1843.890608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1843.917600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1843.943826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1843.981532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1844.034531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1844.077368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1844.140486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1844.172914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1844.204295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1844.228996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1844.258454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1850.922852] LNet: 89226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.929184] LNetError: 89226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.999238] LNet: Removed LNI 192.168.202.59@tcp [ 1852.001498] LNetError: 89226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.017067] LNet: Removed LNI 192.168.202.59@tcp1 [ 1852.699759] Key type .llcrypt unregistered [ 1852.702542] Key type ._llcrypt unregistered [ 1870.016970] Key type ._llcrypt registered [ 1870.018958] Key type .llcrypt registered [ 1871.713324] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 23:31:52 (1787369512) [ 1872.216917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1872.236591] alg: No test for adler32 (adler32-zlib) [ 1873.172877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.233846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1873.262662] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1874.160371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1874.206599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.59@tcp [ 1874.252388] LNetError: 90207:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.59@tcp to recovery [ 1874.274579] LNet: 1 local NIs in recovery (showing 1): 192.168.202.59@tcp [ 1874.395484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.59@tcp [ 1874.497518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1880.326930] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 23:32:01 (1787369521) [ 1880.546565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.596820] LNet: 90796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.606693] LNetError: 90796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.619244] LNet: Removed LNI 192.168.202.59@tcp [ 1880.641262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.722792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1880.757451] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1881.493069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1881.512719] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 1882.362339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.59@tcp [ 1882.382979] LNetError: 91062:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.59@tcp to recovery [ 1882.722169] LNet: There was an unexpected network error while writing to 192.168.202.59: rc = -22 [ 1883.705551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.59@tcp [ 1891.833733] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 23:32:12 (1787369532) [ 1893.564799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg259-server [ 1895.208781] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 23:32:16 (1787369536) [ 1897.229806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg259-server [ 1899.364313] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 23:32:20 (1787369540) [ 1901.632046] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg259-server [ 1903.224634] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 23:32:24 (1787369544) [ 1903.514230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1903.539556] LNet: 92683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.545565] LNetError: 92683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.554485] LNet: Removed LNI 192.168.202.59@tcp [ 1903.565961] LNet: Removed LNI 192.168.202.59@tcp1 [ 1903.596655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.665399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1903.688340] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1904.266218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1904.342531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1904.399230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1904.500270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1904.555712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1904.612626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1911.012969] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 23:32:32 (1787369552) [ 1911.372276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.417821] LNet: 93409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.436171] LNetError: 93409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.447190] LNetError: 93409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1911.468775] LNet: Removed LNI 192.168.202.59@tcp [ 1911.508606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.555814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1911.575171] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1919.256458] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 23:32:40 (1787369560) [ 1919.359274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.391899] LNet: 94103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.410699] LNetError: 94103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.436811] LNet: Removed LNI 192.168.202.59@tcp [ 1919.469784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.560336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1919.585982] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1926.705869] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 23:32:47 (1787369567) [ 1928.410131] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1930.399779] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 23:32:51 (1787369571) [ 1930.899652] LNet: 95148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.905887] LNetError: 95148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.920578] LNet: Removed LNI 192.168.202.59@tcp [ 1931.674196] Key type .llcrypt unregistered [ 1931.677040] Key type ._llcrypt unregistered [ 1939.789228] Key type ._llcrypt registered [ 1939.790663] Key type .llcrypt registered [ 1940.011411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1940.040272] alg: No test for adler32 (adler32-zlib) [ 1941.047064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1941.128153] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1949.051109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 1954.580906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.159@tcp [ 1955.309470] LNet: 95574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.330786] LNetError: 95574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1955.375588] LNet: Removed LNI 192.168.202.59@tcp [ 1956.071810] Key type .llcrypt unregistered [ 1956.083668] Key type ._llcrypt unregistered [ 1969.041498] Key type ._llcrypt registered [ 1969.046433] Key type .llcrypt registered [ 1970.368535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1972.451171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1974.362340] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 23:33:35 (1787369615) [ 1976.178419] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1978.026261] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 23:33:38 (1787369618) [ 1979.900688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg259-server [ 1981.805676] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 23:33:42 (1787369622) [ 1983.645320] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg259-server [ 1985.730293] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 23:33:46 (1787369626) [ 1988.024257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.050669] alg: No test for adler32 (adler32-zlib) [ 1989.286947] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 1989.530926] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 1991.247186] Key type lgssc registered [ 1992.660493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2012.306353] Key type lgssc unregistered [ 2012.598139] LNet: 98755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.611070] LNetError: 98755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2012.644260] LNet: Removed LNI 192.168.202.59@tcp [ 2013.271389] Key type .llcrypt unregistered [ 2013.273465] Key type ._llcrypt unregistered [ 2027.500279] Key type ._llcrypt registered [ 2027.502701] Key type .llcrypt registered [ 2028.575485] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 23:34:29 (1787369669) [ 2028.970878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2028.987286] alg: No test for adler32 (adler32-zlib) [ 2029.925306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.983637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2030.034822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2030.075581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2030.123164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2037.585061] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 23:34:38 (1787369678) [ 2037.821423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2037.850882] LNet: 100172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.885519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.965224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2037.989112] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2038.943693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.959247] LNet: 100316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.965713] LNetError: 100316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.983983] LNet: Removed LNI 192.168.202.59@tcp [ 2039.005959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.104803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2039.152440] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2039.584423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2039.605864] LNetError: 100344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2039.627723] LNet: Removed LNI 192.168.202.59@tcp [ 2039.651368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.698281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2039.718569] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2046.629511] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 23:34:47 (1787369687) [ 2046.786369] LNet: 100927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.790748] LNet: 100927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2046.802851] LNetError: 100927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.819649] LNet: Removed LNI 192.168.202.59@tcp [ 2047.427967] Key type .llcrypt unregistered [ 2047.430834] Key type ._llcrypt unregistered [ 2053.682228] Key type ._llcrypt registered [ 2053.684372] Key type .llcrypt registered [ 2053.900560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2053.918191] alg: No test for adler32 (adler32-zlib) [ 2054.861743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.907374] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2054.909113] LNet: *** cfs_fail_loc=e003, val=100*** [ 2055.561159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2055.577256] LNet: *** cfs_fail_loc=e003, val=100*** [ 2055.579814] LNet: Skipped 2 previous similar messages [ 2055.586325] Ping reply received with mapping: [ 2055.588242] found metadata reply NI[192.168.202.59@tcp] [ 2055.590171] 3b ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2055.597355] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2055.600541] 5a 5a 5a 5a 5a 5a 5a 5a [ 2055.812863] LNet: 101336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.821758] LNetError: 101336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.843154] LNet: Removed LNI 192.168.202.59@tcp [ 2056.567112] Key type .llcrypt unregistered [ 2056.570067] Key type ._llcrypt unregistered [ 2068.543971] Key type ._llcrypt registered [ 2068.546657] Key type .llcrypt registered [ 2069.642458] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 23:35:10 (1787369710) [ 2070.125683] Key type .llcrypt unregistered [ 2070.127529] Key type ._llcrypt unregistered [ 2077.210597] Key type ._llcrypt registered [ 2077.212416] Key type .llcrypt registered [ 2077.345201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2077.357351] alg: No test for adler32 (adler32-zlib) [ 2078.655908] LNetError: 102394:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2079.163444] Key type .llcrypt unregistered [ 2079.165396] Key type ._llcrypt unregistered [ 2085.288131] Key type ._llcrypt registered [ 2085.290755] Key type .llcrypt registered [ 2086.810366] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 23:35:27 (1787369727) [ 2087.375461] Key type .llcrypt unregistered [ 2087.377620] Key type ._llcrypt unregistered [ 2092.509036] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2092.518589] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2095.203810] Key type ._llcrypt registered [ 2095.206588] Key type .llcrypt registered [ 2095.391093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2095.404857] alg: No test for adler32 (adler32-zlib) [ 2096.345181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.411915] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2105.693561] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 23:35:46 (1787369746) [ 2105.855567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.879853] LNet: 104180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.886579] LNetError: 104180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2105.905610] LNet: Removed LNI 10.1.2.3@tcp [ 2105.940496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.489621] Key type .llcrypt unregistered [ 2106.493611] Key type ._llcrypt unregistered [ 2112.714263] Key type ._llcrypt registered [ 2112.723478] Key type .llcrypt registered [ 2112.977075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2113.019965] alg: No test for adler32 (adler32-zlib) [ 2114.036535] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2114.122112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2114.260396] LNet: 104476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2114.277340] LNetError: 104476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.305100] LNet: Removed LNI 192.168.202.59@tcp [ 2115.143475] Key type .llcrypt unregistered [ 2115.148320] Key type ._llcrypt unregistered [ 2127.283902] Key type ._llcrypt registered [ 2127.285917] Key type .llcrypt registered [ 2128.839989] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 23:36:09 (1787369769) [ 2129.563296] Key type .llcrypt unregistered [ 2129.564588] Key type ._llcrypt unregistered [ 2137.181365] Key type ._llcrypt registered [ 2137.190243] Key type .llcrypt registered [ 2137.584361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2137.599303] alg: No test for adler32 (adler32-zlib) [ 2146.356868] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 23:36:27 (1787369787) [ 2146.485217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.560862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.678381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2146.704744] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2147.559331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2147.581670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2147.607174] LNet: Removed LNI 192.168.202.59@tcp [ 2148.130580] LNet: 106234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.610748] Key type .llcrypt unregistered [ 2148.615505] Key type ._llcrypt unregistered [ 2160.551810] Key type ._llcrypt registered [ 2160.556873] Key type .llcrypt registered [ 2162.326205] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 23:36:42 (1787369802) [ 2162.863987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2162.887806] alg: No test for adler32 (adler32-zlib) [ 2163.861535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.967256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2163.997291] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2165.203486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2165.273149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.59@tcp [ 2165.307277] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2165.315663] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2165.323175] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2165, 0, 0 [ 2165.329217] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2165.332873] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2165.338868] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2165.344140] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2165, 0, 0 [ 2165.350263] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2165.354395] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2165.360873] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2165.364744] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2165, 0, 2215 [ 2165.377888] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2165.381657] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2165.385902] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2165.390661] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2165, 0, 2215 [ 2165.401367] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2173.343408] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 23:36:54 (1787369814) [ 2173.964574] LNet: 107774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2173.973473] LNetError: 107774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2173.984294] LNet: Removed LNI 192.168.202.59@tcp [ 2174.728161] Key type .llcrypt unregistered [ 2174.730196] Key type ._llcrypt unregistered [ 2182.172270] Key type ._llcrypt registered [ 2182.173944] Key type .llcrypt registered [ 2182.629698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2182.675874] alg: No test for adler32 (adler32-zlib) [ 2183.658751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2183.694315] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2191.965824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 2200.950580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.036508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2201.070768] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 2202.015740] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2202.076469] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2202.081846] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2202.084234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2204.493497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2204.519461] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2215.009183] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 23:37:36 (1787369856) [ 2215.353467] LNet: 109290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.360223] LNetError: 109290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.367755] LNet: Removed LNI 192.168.202.59@tcp [ 2216.415220] LNet: Removed LNI 192.168.202.59@tcp1 [ 2216.419160] LNetError: 109290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2216.434551] LNetError: 109290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2216.462350] LNet: Removed LNI 10.1.2.3@tcp99 [ 2217.289375] Key type .llcrypt unregistered [ 2217.291824] Key type ._llcrypt unregistered [ 2225.915497] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2225.919084] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2228.566554] Key type ._llcrypt registered [ 2228.568735] Key type .llcrypt registered [ 2228.802584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2228.830912] alg: No test for adler32 (adler32-zlib) [ 2229.897609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.017415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2230.090489] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2231.224509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2231.248410] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2233.847471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2233.898584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2233.944661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2233.964766] LNetError: 110040:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2234.014383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2234.066819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2234.736762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2234.831739] LNet: 110060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.841198] LNetError: 110060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2234.849249] LNet: Removed LNI 10.1.2.3@tcp [ 2235.887690] LNet: Removed LNI 192.168.202.59@tcp [ 2236.629734] Key type .llcrypt unregistered [ 2236.631459] Key type ._llcrypt unregistered [ 2250.235823] Key type ._llcrypt registered [ 2250.241172] Key type .llcrypt registered [ 2251.944197] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 23:38:12 (1787369892) [ 2252.446461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.473487] alg: No test for adler32 (adler32-zlib) [ 2253.410305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.515239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2253.555624] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2254.379209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2254.475333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg259-client@tcp [ 2261.476337] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 23:38:22 (1787369902) [ 2261.585058] LNet: 111600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.593313] LNetError: 111600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2261.612221] LNet: Removed LNI 192.168.202.59@tcp [ 2262.390403] Key type .llcrypt unregistered [ 2262.393246] Key type ._llcrypt unregistered [ 2268.239187] Key type ._llcrypt registered [ 2268.241157] Key type .llcrypt registered [ 2268.461532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2268.495501] alg: No test for adler32 (adler32-zlib) [ 2269.426935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2269.447306] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2269.472938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2277.389426] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 23:38:38 (1787369918) [ 2278.853889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2280.369871] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 23:38:41 (1787369921) [ 2282.183577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2284.022962] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 23:38:44 (1787369924) [ 2284.137796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.192393] LNet: 113025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.201545] LNetError: 113025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.238971] LNet: Removed LNI 192.168.202.59@tcp [ 2284.259932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.393441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2284.429807] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2285.145171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2285.190179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2292.760608] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 23:38:53 (1787369933) [ 2293.216288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.260637] LNet: 113742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.275877] LNetError: 113742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.298668] LNet: Removed LNI 192.168.202.59@tcp [ 2293.371490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.449309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2293.519074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2294.431513] Key type .llcrypt unregistered [ 2294.436698] Key type ._llcrypt unregistered [ 2308.809714] Key type ._llcrypt registered [ 2308.810877] Key type .llcrypt registered [ 2310.121911] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 23:39:11 (1787369951) [ 2310.892614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.907773] alg: No test for adler32 (adler32-zlib) [ 2311.945964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.996523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2312.017635] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2312.902687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2312.990287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2320.213240] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 23:39:20 (1787369960) [ 2320.418653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2320.462729] LNet: 115300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.479172] LNetError: 115300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.516199] LNet: Removed LNI 192.168.202.59@tcp [ 2320.592033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.706556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2320.755952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2328.243614] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 23:39:29 (1787369969) [ 2328.542287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.567893] LNet: 115884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.584690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.650266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2329.051794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2329.102613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2329.174430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2329.622469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2329.662570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2329.759290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.799181] LNet: 115917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.846539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.932675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2330.583891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2330.637500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2330.689598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2331.227935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2331.292671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2338.022252] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 23:39:38 (1787369978) [ 2338.215403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2338.260943] LNet: 116517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.308320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.366287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2338.388262] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2339.060982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2339.088413] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2340.125375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2340.212779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2340.446993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.468881] LNetError: 116797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.475872] LNet: Removed LNI 192.168.202.59@tcp [ 2340.500394] LNet: Removed LNI 192.168.202.59@tcp2 [ 2340.527594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.620573] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2340.626805] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2340.631479] LNet: 116803:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2341.230907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2341.542244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.558607] LNetError: 116840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2341.564225] LNetError: 116840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2342.623249] LNet: Removed LNI 192.168.202.59@tcp [ 2342.634252] LNet: Removed LNI 192.168.202.59@tcp2 [ 2342.666459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.753483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2342.772608] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2343.498574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2343.522455] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2344.391171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2344.438257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2344.510973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2344.687381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.703602] LNet: 117125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.707907] LNet: 117125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2344.716168] LNetError: 117125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.720893] LNetError: 117125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2345.759232] LNet: Removed LNI 192.168.202.59@tcp [ 2346.790801] LNet: Removed LNI 192.168.202.59@tcp2 [ 2346.833844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.934780] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2346.939457] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2346.948092] LNet: 117131:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.670566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2347.897866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.913464] LNet: Removed LNI 192.168.202.59@tcp [ 2348.966939] LNet: Removed LNI 192.168.202.59@tcp2 [ 2349.022265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.088692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.113468] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2349.807585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2349.824530] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2350.770355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2350.833903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2351.031446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.052962] LNetError: 117448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.056886] LNetError: 117448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2351.062914] LNet: Removed LNI 192.168.202.59@tcp [ 2351.071793] LNet: Removed LNI 192.168.202.59@tcp2 [ 2351.092532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.146421] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2351.152408] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2351.159772] LNet: 117454:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2351.670503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2351.842430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.927154] LNet: Removed LNI 192.168.202.59@tcp [ 2352.953215] LNet: Removed LNI 192.168.202.59@tcp2 [ 2353.046264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.149172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2353.183511] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2353.937457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2353.952574] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2354.740561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2354.785630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2354.880929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2355.183868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2355.214479] LNet: 117775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.221864] LNet: 117775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2356.255215] LNet: Removed LNI 192.168.202.59@tcp [ 2357.293848] LNet: Removed LNI 192.168.202.59@tcp2 [ 2357.324448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.388961] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2357.408150] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2357.422763] LNet: 117782:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2358.067093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2358.326478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.352826] LNet: Removed LNI 192.168.202.59@tcp [ 2359.400184] LNet: Removed LNI 192.168.202.59@tcp2 [ 2359.452670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.515562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2359.548843] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2360.435323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2360.472904] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2361.245939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2361.275686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2361.320435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2361.734263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.761444] LNetError: 118115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.768223] LNetError: 118115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2361.779707] LNet: Removed LNI 192.168.202.59@tcp [ 2361.793757] LNet: Removed LNI 192.168.202.59@tcp2 [ 2361.841121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.949779] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2361.956576] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2361.969720] LNet: 118121:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2362.607797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2363.000598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2364.063228] LNet: Removed LNI 192.168.202.59@tcp [ 2364.090166] LNet: Removed LNI 192.168.202.59@tcp2 [ 2364.140686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.247490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2364.272546] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2364.941412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2364.975604] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2365.896850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2365.948636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2365.971456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2366.044948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2366.444207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2367.524614] LNet: Removed LNI 192.168.202.59@tcp [ 2368.552926] LNet: Removed LNI 192.168.202.59@tcp2 [ 2368.586794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.654620] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2368.658790] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2369.246984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2376.759601] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2378.515245] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 23:40:19 (1787370019) [ 2378.879819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2378.905901] LNet: 119241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.924329] LNet: 119241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2378.933385] LNetError: 119241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.943947] LNetError: 119241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2378.955579] LNet: Removed LNI 192.168.202.59@tcp [ 2380.007545] LNet: Removed LNI 192.168.202.59@tcp2 [ 2380.023752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.407725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2380.428600] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2381.353705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2381.389977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2382.140329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2382.171377] LNet: Removed LNI 10.1.2.3@tcp [ 2382.189664] LNet: Removed LNI 192.168.202.59@tcp [ 2382.217810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.275592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2382.301553] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2382.309037] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2382.956524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2382.994874] LNet: Removed LNI 192.168.202.59@tcp [ 2383.021030] LNet: Removed LNI 10.1.2.3@tcp [ 2391.035537] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 23:40:31 (1787370031) [ 2391.183953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2391.257603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2391.361312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2391.397571] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2392.525682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2392.578696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2392.655650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2392.735362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2392.796680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2392.856219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2392.893760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2392.922868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2392.965674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2392.993985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2393.043419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.59@tcp [ 2393.192493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2400.567852] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 23:40:41 (1787370041) [ 2400.661608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.703603] LNet: Removed LNI 192.168.202.59@tcp [ 2401.797898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.043369] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2402.346598] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2402.349562] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2402.408492] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2404.925910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2404.957297] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2405.692376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2405.715710] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2407.876885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2407.948739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2408.037421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.59@tcp -m PUT -e local_timeout -r 1 [ 2408.114541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2408.210995] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2408.223220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2408.653704] LNetError: 120914:0:(lib-move.c:4801:lnet_parse()) 192.168.202.59@tcp, src 192.168.202.59@tcp: Dropping PUT (error -108 looking up sender) [ 2409.250895] LNet: Removed LNI 10.1.2.3@tcp [ 2410.280249] LNet: Removed LNI 192.168.202.59@tcp [ 2410.728710] Key type .llcrypt unregistered [ 2410.730790] Key type ._llcrypt unregistered [ 2423.699878] Key type ._llcrypt registered [ 2423.702835] Key type .llcrypt registered [ 2425.747154] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 23:41:05 (1787370065) [ 2426.288178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2426.314633] alg: No test for adler32 (adler32-zlib) [ 2427.388163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2427.993792] LNet: 122231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.103565] Key type .llcrypt unregistered [ 2429.110700] Key type ._llcrypt unregistered [ 2438.437901] Key type ._llcrypt registered [ 2438.439370] Key type .llcrypt registered [ 2438.616954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2438.635367] alg: No test for adler32 (adler32-zlib) [ 2439.829843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2439.891124] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2449.224495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 2449.541240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2450.407496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.159@tcp [ 2450.682774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2450.711202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2450.753158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2450.807705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2450.844913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2450.884839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2450.948607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2451.011993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2451.035362] LNet: 122685:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2451.072582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2451.118733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2451.167668] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.159@tcp [ 2451.180918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2451.216477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2451.249661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2451.293150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2452.193757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2452.557231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2452.618839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2452.702330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2452.763128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2452.812581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2452.875435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2452.933994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2452.998331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2453.020785] LNet: 122803:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2453.063291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2453.108367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2453.144573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2453.187929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2453.230687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2453.266884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2454.023409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.159@tcp [ 2454.284080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2454.318605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2454.351544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2454.385262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2454.435844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2454.503379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2454.565324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2454.635518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2454.663185] LNet: 122920:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2454.701460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2454.782455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2454.844644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2454.897407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2454.995523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2455.079730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2455.881424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2456.298771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2456.373373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2456.421631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2456.476903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2456.520468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2456.567996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2456.661071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2456.734359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2456.784369] LNet: 123037:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2456.831196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2456.884734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2456.951241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2457.042352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2457.096685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2457.181145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2457.846955] LNet: 123087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.855417] LNetError: 123087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.869425] LNet: Removed LNI 192.168.202.59@tcp [ 2458.320724] Key type .llcrypt unregistered [ 2458.325815] Key type ._llcrypt unregistered [ 2479.287755] Key type ._llcrypt registered [ 2479.292029] Key type .llcrypt registered [ 2480.943406] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 23:42:01 (1787370121) [ 2481.461057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2481.479739] alg: No test for adler32 (adler32-zlib) [ 2482.400383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2482.908423] LNet: 123975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2483.741678] Key type .llcrypt unregistered [ 2483.746690] Key type ._llcrypt unregistered [ 2490.239592] Key type ._llcrypt registered [ 2490.243693] Key type .llcrypt registered [ 2490.741340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2490.793467] alg: No test for adler32 (adler32-zlib) [ 2491.783738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2491.814109] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2499.694949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.159@tcp [ 2510.882243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2510.948888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2511.003227] LNet: Added LNI 192.168.202.59@tcp1 [8/256/0/180] [ 2512.084196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2512.892908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.159@tcp [ 2513.240780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2513.282620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2513.318454] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.159@tcp [ 2513.336418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2513.366789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2513.400750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2513.446974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2513.516420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2513.594977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2513.625388] LNet: 124639:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2513.651090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2513.741755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2513.841648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2513.930801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2513.988170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2514.050388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2514.922802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2515.269853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2515.304963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2515.346545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2515.402831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2515.455595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2515.514113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2515.563251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2515.598542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2515.627769] LNet: 124756:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2515.649787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2515.688594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2515.738424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2515.783885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2515.833333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2515.875837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2516.610152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.159@tcp [ 2516.845291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2516.880001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2516.908622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2516.938854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2516.969128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2516.998192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2517.030703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2517.064435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2517.084417] LNet: 124873:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2517.101571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2517.131666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2517.169215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2517.206088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2517.240751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2517.282821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2518.081751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2518.381764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2518.445261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2518.496752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2518.548313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2518.587714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2518.636930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2518.663958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2518.691657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2518.719159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2518.780398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2518.830535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2518.882974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2518.956903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2519.021750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2524.081720] LNet: 125075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2524.088572] LNetError: 125075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2525.151184] LNet: Removed LNI 192.168.202.59@tcp [ 2525.159389] LNetError: 125075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2525.176067] LNet: Removed LNI 192.168.202.59@tcp1 [ 2525.614927] Key type .llcrypt unregistered [ 2525.619041] Key type ._llcrypt unregistered [ 2546.017783] Key type ._llcrypt registered [ 2546.021069] Key type .llcrypt registered [ 2547.299847] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 23:43:08 (1787370188) [ 2547.886740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2547.901483] alg: No test for adler32 (adler32-zlib) [ 2548.868609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2548.944616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2548.975510] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2549.701377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2549.722497] LNet: Added LNI 192.168.202.59@tcp2 [8/256/0/180] [ 2550.824315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.59@tcp [ 2550.903523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2551.967796] LNet: Removed LNI 192.168.202.59@tcp [ 2559.099498] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 23:43:19 (1787370199) [ 2560.915538] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2562.895328] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 23:43:23 (1787370203) [ 2572.041698] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 23:43:32 (1787370212) [ 2580.942662] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 23:43:41 (1787370221) [ 2589.969161] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 23:43:50 (1787370230) [ 2598.597224] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 23:43:59 (1787370239) [ 2607.000549] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 23:44:07 (1787370247) [ 2614.770776] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 23:44:16 (1787370256) [ 2623.472747] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 23:44:24 (1787370264) [ 2632.334953] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 23:44:33 (1787370273) [ 2639.652344] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 23:44:40 (1787370280) [ 2646.995950] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 23:44:47 (1787370287) [ 2655.168638] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 23:44:56 (1787370296) [ 2662.052523] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 23:45:03 (1787370303) [ 2670.714925] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 23:45:11 (1787370311) [ 2680.398222] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 23:45:20 (1787370320) [ 2690.360338] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 23:45:30 (1787370330) [ 2698.517042] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 23:45:39 (1787370339) [ 2705.942680] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 23:45:46 (1787370346) [ 2706.073440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2706.114301] LNet: 136766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2706.130235] LNetError: 136766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2706.172899] LNet: Removed LNI 192.168.202.59@tcp2 [ 2706.235590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2706.371632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2706.388942] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2707.656941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2707.696811] LNet: 136914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2707.701908] LNetError: 136914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2707.725385] LNet: Removed LNI 192.168.202.59@tcp [ 2707.784342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2707.839538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2707.868781] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2714.246972] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 23:45:55 (1787370355) [ 2721.108691] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 23:46:01 (1787370361) [ 2729.582279] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 23:46:10 (1787370370) [ 2737.898811] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 23:46:18 (1787370378) [ 2746.141562] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 23:46:26 (1787370386) [ 2753.024903] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 23:46:34 (1787370394) [ 2760.628806] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 23:46:41 (1787370401) [ 2768.869225] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 23:46:49 (1787370409) [ 2776.497376] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 23:46:57 (1787370417) [ 2784.239577] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 23:47:05 (1787370425) [ 2791.420650] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 23:47:12 (1787370432) [ 2798.477638] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 23:47:19 (1787370439) [ 2805.840831] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2549 sec ========== 23:47:26 (1787370446) [ 2806.578960] LNet: 144382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2806.585724] LNetError: 144382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2806.601249] LNet: Removed LNI 192.168.202.59@tcp [ 2807.500707] Key type .llcrypt unregistered [ 2807.503577] Key type ._llcrypt unregistered [ 2820.933935] Key type ._llcrypt registered [ 2820.945795] Key type .llcrypt registered [ 2821.562052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2821.580851] alg: No test for adler32 (adler32-zlib) [ 2822.618996] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 2822.795492] LNet: Added LNI 192.168.202.59@tcp [8/256/0/180] [ 2824.479218] Key type lgssc registered [ 2825.907596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2887.649746] Lustre: Mounted lustre-client - version 2.17.57_45_gb800aec [ 2892.306882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2901.171715] Lustre: Unmounted lustre-client [ 2937.554232] Key type lgssc unregistered [ 2937.825665] LNet: 147342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2937.831908] LNetError: 147342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2937.846492] LNet: Removed LNI 192.168.202.59@tcp [ 2938.483208] Key type .llcrypt unregistered [ 2938.485294] Key type ._llcrypt unregistered