[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 601201733 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.004293] x2apic enabled [ 0.006006] Switched APIC routing to physical x2apic. [ 0.007023] kvm-guest: setup PV IPIs [ 0.016000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.016000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.016021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.017029] pid_max: default: 32768 minimum: 301 [ 0.018159] LSM: Security Framework initializing [ 0.019196] Yama: becoming mindful. [ 0.020095] SELinux: Initializing. [ 0.021155] *** VALIDATE selinux *** [ 0.032255] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.038922] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.039186] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.041117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.042252] *** VALIDATE tmpfs *** [ 0.044665] *** VALIDATE proc *** [ 0.045616] *** VALIDATE cgroup *** [ 0.046016] *** VALIDATE cgroup2 *** [ 0.047000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.049008] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.050012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.051000] Spectre V2 : User space: Vulnerable [ 0.051078] Speculative Store Bypass: Vulnerable [ 0.054780] debug: unmapping init [mem 0xffffffffb8a59000-0xffffffffb8a60fff] [ 0.056319] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.058120] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.059028] ... version: 2 [ 0.060010] ... bit width: 48 [ 0.061009] ... generic registers: 4 [ 0.062010] ... value mask: 0000ffffffffffff [ 0.063019] ... max period: 00007fffffffffff [ 0.064017] ... fixed-purpose events: 3 [ 0.065016] ... event mask: 000000070000000f [ 0.066338] rcu: Hierarchical SRCU implementation. [ 0.068617] smp: Bringing up secondary CPUs ... [ 0.069664] x86: Booting SMP configuration: [ 0.070028] .... node #0, CPUs: #1 #2 #3 [ 0.081163] smp: Brought up 1 node, 4 CPUs [ 0.083014] smpboot: Max logical packages: 1 [ 0.084014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130587] node 0 deferred pages initialised in 45ms [ 0.135699] devtmpfs: initialized [ 0.137378] x86/mm: Memory block size: 128MB [ 0.143173] gcov: version magic: 0x41383552 [ 0.146892] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148239] pinctrl core: initialized pinctrl subsystem [ 0.149713] [ 0.150016] ************************************************************* [ 0.151019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152021] ** ** [ 0.153023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154022] ** ** [ 0.155017] ** This means that this kernel is built to expose internal ** [ 0.156014] ** IOMMU data structures, which may compromise security on ** [ 0.157032] ** your system. ** [ 0.158019] ** ** [ 0.159013] ** If you see this message and you are not debugging the ** [ 0.160020] ** kernel, report this immediately to your vendor! ** [ 0.161018] ** ** [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163029] ************************************************************* [ 0.165104] NET: Registered protocol family 16 [ 0.166677] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169816] cpuidle: using governor menu [ 0.171172] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174820] PCI: Using configuration type 1 for base access [ 0.177128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196053] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198538] cryptd: max_cpu_qlen set to 1000 [ 0.199413] ACPI: Added _OSI(Module Device) [ 0.200000] ACPI: Added _OSI(Processor Device) [ 0.203025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211529] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220603] ACPI: Interpreter enabled [ 0.221069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.222013] ACPI: Using IOAPIC for interrupt routing [ 0.223127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224589] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236238] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237895] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.238021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.239096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.244094] acpiphp: Slot [2] registered [ 0.245166] acpiphp: Slot [5] registered [ 0.246164] acpiphp: Slot [6] registered [ 0.247103] acpiphp: Slot [3] registered [ 0.248781] acpiphp: Slot [4] registered [ 0.249177] acpiphp: Slot [7] registered [ 0.251068] acpiphp: Slot [8] registered [ 0.252352] acpiphp: Slot [9] registered [ 0.254112] acpiphp: Slot [10] registered [ 0.255276] acpiphp: Slot [11] registered [ 0.256127] acpiphp: Slot [12] registered [ 0.257107] acpiphp: Slot [13] registered [ 0.258147] acpiphp: Slot [14] registered [ 0.259106] acpiphp: Slot [15] registered [ 0.260301] acpiphp: Slot [16] registered [ 0.261142] acpiphp: Slot [17] registered [ 0.262141] acpiphp: Slot [18] registered [ 0.264000] acpiphp: Slot [19] registered [ 0.264144] acpiphp: Slot [20] registered [ 0.265143] acpiphp: Slot [21] registered [ 0.267024] acpiphp: Slot [22] registered [ 0.268134] acpiphp: Slot [23] registered [ 0.269104] acpiphp: Slot [24] registered [ 0.270129] acpiphp: Slot [25] registered [ 0.271154] acpiphp: Slot [26] registered [ 0.272142] acpiphp: Slot [27] registered [ 0.273895] acpiphp: Slot [28] registered [ 0.274133] acpiphp: Slot [29] registered [ 0.275126] acpiphp: Slot [30] registered [ 0.276384] acpiphp: Slot [31] registered [ 0.277077] PCI host bridge to bus 0000:00 [ 0.278037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.279045] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.280028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.281022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.282025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.283028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.284204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.285997] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.288346] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.296019] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.300081] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.301027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.302021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.303033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.304910] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.308780] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.313097] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.318708] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.328015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.345019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353821] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.363303] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.373017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.419019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.433980] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.455000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.477018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.489555] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.490000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.494403] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.496333] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.498199] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.502120] iommu: Default domain type: Passthrough [ 0.505640] SCSI subsystem initialized [ 0.506154] ACPI: bus type USB registered [ 0.507084] usbcore: registered new interface driver usbfs [ 0.508072] usbcore: registered new interface driver hub [ 0.510076] usbcore: registered new device driver usb [ 0.512176] pps_core: LinuxPPS API ver. 1 registered [ 0.514010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.517061] PTP clock support registered [ 0.518316] EDAC MC: Ver: 3.0.0 [ 0.519315] PCI: Using ACPI for IRQ routing [ 0.525294] NetLabel: Initializing [ 0.526000] NetLabel: domain hash size = 128 [ 0.526000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.529080] NetLabel: unlabeled traffic allowed by default [ 0.530366] vgaarb: loaded [ 0.533174] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.535012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.541000] clocksource: Switched to clocksource kvm-clock [ 0.676085] VFS: Disk quotas dquot_6.6.0 [ 0.678487] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.681469] *** VALIDATE ramfs *** [ 0.682636] *** VALIDATE hugetlbfs *** [ 0.684768] pnp: PnP ACPI init [ 0.687947] pnp: PnP ACPI: found 6 devices [ 0.737764] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.750191] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.755144] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.759195] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.763423] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.767145] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.771318] NET: Registered protocol family 2 [ 0.774911] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.781578] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.786680] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.793662] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.798315] TCP: Hash tables configured (established 65536 bind 65536) [ 0.802563] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.807071] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.810626] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.814599] NET: Registered protocol family 1 [ 0.816994] RPC: Registered named UNIX socket transport module. [ 0.819290] RPC: Registered udp transport module. [ 0.820840] RPC: Registered tcp transport module. [ 0.822444] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.824341] NET: Registered protocol family 44 [ 0.827046] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.833119] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.836015] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.838892] PCI: CLS 0 bytes, default 64 [ 0.842108] Unpacking initramfs... [ 3.532947] debug: unmapping init [mem 0xffff8f8c7cc64000-0xffff8f8c7ffcffff] [ 3.537425] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.540036] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.543815] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.763111] Initialise system trusted keyrings [ 4.766484] Key type blacklist registered [ 4.773283] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.811501] zbud: loaded [ 4.825268] *** VALIDATE nfs *** [ 4.831315] *** VALIDATE nfs4 *** [ 4.844598] pstore: using deflate compression [ 4.851406] Platform Keyring initialized [ 5.059595] NET: Registered protocol family 38 [ 5.062015] Key type asymmetric registered [ 5.069994] Asymmetric key parser 'x509' registered [ 5.077677] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.086206] io scheduler mq-deadline registered [ 5.088021] io scheduler kyber registered [ 5.097555] io scheduler bfq registered [ 5.109387] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.119645] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.133226] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.145751] ACPI: Power Button [PWRF] [ 5.179784] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.209274] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.241028] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.285544] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.320385] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.343945] Non-volatile memory driver v1.3 [ 5.346846] Linux agpgart interface v0.103 [ 5.435859] virtio_blk virtio1: [vda] 150632 512-byte logical blocks (77.1 MB/73.6 MiB) [ 5.439075] vda: detected capacity change from 0 to 77123584 [ 5.458419] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.462575] vdb: detected capacity change from 0 to 1073741824 [ 5.477519] libphy: Fixed MDIO Bus: probed [ 5.490133] usbcore: registered new interface driver usbserial_generic [ 5.492712] usbserial: USB Serial support registered for generic [ 5.495028] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.502467] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.509491] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.521871] mousedev: PS/2 mouse device common for all mice [ 5.529505] rtc_cmos 00:05: RTC can wake from S4 [ 5.539281] rtc_cmos 00:05: registered as rtc0 [ 5.540862] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.547097] intel_pstate: CPU model not supported [ 5.551567] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.566456] hid: raw HID events driver (C) Jiri Kosina [ 5.568183] usbcore: registered new interface driver usbhid [ 5.575211] usbhid: USB HID core driver [ 5.577056] drop_monitor: Initializing network drop monitor service [ 5.579221] Initializing XFRM netlink socket [ 5.584407] NET: Registered protocol family 10 [ 5.585590] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.593763] Segment Routing with IPv6 [ 5.602066] NET: Registered protocol family 17 [ 5.607854] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.610053] mpls_gso: MPLS GSO support [ 5.632459] RAS: Correctable Errors collector initialized. [ 5.636554] AVX version of gcm_enc/dec engaged. [ 5.640304] AES CTR mode by8 optimization enabled [ 5.791393] sched_clock: Marking stable (5791372283, 0)->(7370800128, -1579427845) [ 5.796695] registered taskstats version 1 [ 5.799388] Loading compiled-in X.509 certificates [ 5.802593] zswap: loaded using pool lzo/zbud [ 5.841278] Key type big_key registered [ 5.875598] Key type encrypted registered [ 5.876983] ima: No TPM chip found, activating TPM-bypass! [ 5.882589] ima: Allocated hash algorithm: sha1 [ 5.887317] ima: No architecture policies found [ 5.891524] evm: Initialising EVM extended attributes: [ 5.894440] evm: security.selinux [ 5.896484] evm: security.ima [ 5.898372] evm: security.capability [ 5.901840] evm: HMAC attrs: 0x1 [ 5.919203] rtc_cmos 00:05: setting system clock to 2026-09-29 16:21:11 UTC (1790698871) [ 5.930699] debug: unmapping init [mem 0xffffffffb9a03000-0xffffffffb9bfffff] [ 5.939575] debug: unmapping init [mem 0xffffffffb8782000-0xffffffffb8a58fff] [ 5.950136] Write protecting the kernel read-only data: 28672k [ 5.957079] debug: unmapping init [mem 0xffffffffb6e03000-0xffffffffb6ffffff] [ 5.963585] debug: unmapping init [mem 0xffffffffb7714000-0xffffffffb77fffff] [ 6.030882] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.055582] systemd[1]: Detected virtualization kvm. [ 6.057727] systemd[1]: Detected architecture x86-64. [ 6.060231] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.090823] systemd[1]: No hostname configured. [ 6.092527] systemd[1]: Set hostname to . [ 6.094640] random: systemd: uninitialized urandom read (16 bytes read) [ 6.100300] systemd[1]: Initializing machine ID from random generator. [ 6.489362] random: systemd: uninitialized urandom read (16 bytes read) [ 6.492585] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 6.505588] random: systemd: uninitialized urandom read (16 bytes read) [ 6.509807] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.514759] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ 7.927191] hrtimer: interrupt took 4265821 ns [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.549403] device-mapper: uevent: version 1.0.3 [ 8.553667] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.323424] virtio_net virtio0 ens2: renamed from eth0 [ 10.344394] random: fast init done [ 10.693501] scsi host0: ata_piix [ 10.723926] scsi host1: ata_piix [ 10.731942] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 10.744820] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.429277] random: crng init done [ 15.431991] random: 7 urandom warning(s) missed due to ratelimiting [ 17.596740] dracut-initqueue[575]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.811536] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.030979] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.991967] SELinux: Disabled at runtime. [ 24.153656] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.171110] systemd[1]: Detected virtualization kvm. [ 24.174151] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.475620] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.486692] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.514991] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.536405] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.544626] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.558796] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.598260] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ 25.956608] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 26.843969] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.812670] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.869974] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.177517] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.270457] EDAC sbridge: Ver: 1.1.2 [ 31.693533] Key type dns_resolver registered [ 32.061619] NFS: Registering the id_resolver key type [ 32.064398] Key type id_resolver registered [ 32.066384] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg457-client login: [ 91.425928] libcfs: loading out-of-tree module taints kernel. [ 91.742911] Key type ._llcrypt registered [ 91.785914] Key type .llcrypt registered [ 92.438344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 92.451401] alg: No test for adler32 (adler32-zlib) [ 93.790468] Lustre: Lustre: Build Version: 2.17.59_2_ga235f42 [ 94.773849] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 96.559564] Key type lgssc registered [ 97.969648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 275.284506] Lustre: Mounted lustre-client - version 2.17.59_2_ga235f42 [ 279.746888] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 299.426733] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing check_logdir /tmp/testlogs/ [ 301.028437] Lustre: lustre-OST0000-osc-ffff8f8cc9f40800: disconnect after 24s idle [ 304.862736] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing yml_node [ 308.639392] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 311.439389] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 313.848496] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 315.267491] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 29 12:26:19 EDT 2026 [ 329.945427] Lustre: Unmounted lustre-client [ 381.782653] Key type lgssc unregistered [ 381.996860] LNet: 9213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.004384] LNetError: 9213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 382.034072] LNet: Removed LNI 192.168.204.57@tcp [ 382.628676] Key type .llcrypt unregistered [ 382.631094] Key type ._llcrypt unregistered [ 411.624739] Key type ._llcrypt registered [ 411.627801] Key type .llcrypt registered [ 411.840978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 411.858644] alg: No test for adler32 (adler32-zlib) [ 412.740849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 412.797913] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 412.812750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 416.327415] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 416.510037] LNet: 10151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.540561] LNetError: 10151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 416.579187] LNet: Removed LNI 192.168.204.57@tcp [ 417.116981] Key type .llcrypt unregistered [ 417.121831] Key type ._llcrypt unregistered [ 423.929715] Key type ._llcrypt registered [ 423.933981] Key type .llcrypt registered [ 425.195795] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:28:09 (1790699289) [ 425.700435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 425.710560] alg: No test for adler32 (adler32-zlib) [ 426.645127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.722933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.751643] LNet: 10568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.788140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.871829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 426.895118] LNet: 10575:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 433.577162] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:28:17 (1790699297) [ 433.672442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.690477] LNet: 11156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.716603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.815918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 440.838095] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:28:24 (1790699304) [ 440.938446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.986589] LNet: 11749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.023919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.141811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 449.217943] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:28:32 (1790699312) [ 449.329197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.374117] LNet: 12343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.414121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.570528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 455.897601] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:28:40 (1790699320) [ 456.184456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.216387] LNet: 12941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.251705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.348149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 456.977694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 464.459062] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:28:48 (1790699328) [ 464.581488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.617070] LNet: 13534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.649242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.758679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 465.316858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 465.935196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 471.980817] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:28:56 (1790699336) [ 472.133655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.187178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.261554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 478.362349] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:29:02 (1790699342) [ 478.454118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.513441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.611573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 478.643213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 479.126437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 479.198637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 479.608871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 479.633647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 480.002516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 480.028591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 480.292298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 480.345476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 480.717972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 480.744643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 481.062735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 481.094051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 481.571058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 481.628691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 481.917143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 481.962549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 482.223349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 482.248746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 482.513552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 482.564995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 483.054513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 483.122247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 483.635459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 483.681470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 489.338787] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:29:13 (1790699353) [ 489.453688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.478884] LNet: 15421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.487992] LNet: 15421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 489.522648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.618865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 489.663910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 495.767894] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:29:19 (1790699359) [ 495.861688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.910960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.021752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 496.056172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 502.947391] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:29:26 (1790699366) [ 503.045533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.100857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.173838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 503.216721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 510.978958] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:29:34 (1790699374) [ 511.195637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.252599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.362922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 511.434339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 519.312276] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:29:43 (1790699383) [ 519.436217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.508025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.610882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 519.662360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 526.584185] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:29:50 (1790699390) [ 526.746566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.761358] LNet: 18399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.765774] LNet: 18399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 526.780761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.860568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 526.919730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 533.320664] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:29:57 (1790699397) [ 533.548365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.577306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.644443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 534.083973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 540.568779] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:30:04 (1790699404) [ 540.768736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.836610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.062228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 541.234936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 546.949364] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:30:11 (1790699411) [ 547.033864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.085737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.191906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 547.373607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 552.761750] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:30:17 (1790699417) [ 552.899197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.953109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.457796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 553.685651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 559.764217] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:30:24 (1790699424) [ 559.954312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.983311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.034214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 560.080574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 566.381785] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:30:30 (1790699430) [ 566.568273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.615832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.682279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 566.738867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 572.332989] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:30:36 (1790699436) [ 572.675374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.751873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.867416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 579.396584] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:30:43 (1790699443) [ 579.629109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.728660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.826573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 586.476859] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:30:50 (1790699450) [ 586.706567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.749603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.828291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 587.215787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 593.305617] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:30:57 (1790699457) [ 593.437376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.482150] LNet: 24504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.487036] LNet: 24504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 593.524784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.588076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 593.630233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 599.260681] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:31:03 (1790699463) [ 599.329378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.373690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.442294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 599.480842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 605.517891] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:31:10 (1790699470) [ 605.722545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.753355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.800537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 605.834624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 611.043616] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:31:15 (1790699475) [ 611.220437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.265132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.318259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 611.347860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 617.926039] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:31:22 (1790699482) [ 618.143129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.180531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.230958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 618.279638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 618.292969] LNetError: 26895:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 618.314111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 618.350756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 624.028783] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:31:28 (1790699488) [ 624.325690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.375927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.500262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DXmm8LJcbe [ 624.621350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DXmm8LJcbe [ 624.681409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DXmm8LJcbe [ 631.635434] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:31:35 (1790699495) [ 631.784853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.856402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.935920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 632.008472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 632.082781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 639.711015] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:31:44 (1790699504) [ 639.830247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.886351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.999430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.066058] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 641.925980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 643.099958] Lustre: Lustre: Build Version: 2.17.59_2_ga235f42 [ 644.959513] Key type lgssc registered [ 646.129362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 649.731938] LNetError: 28675:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.57@tcp99 has no available nets [ 651.589583] Key type lgssc unregistered [ 651.941888] LNetError: 29822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.953636] LNet: Removed LNI 192.168.204.57@tcp [ 652.548150] Key type .llcrypt unregistered [ 652.554124] Key type ._llcrypt unregistered [ 660.389106] Key type ._llcrypt registered [ 660.391524] Key type .llcrypt registered [ 661.646667] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:32:05 (1790699525) [ 662.259625] Key type .llcrypt unregistered [ 662.262542] Key type ._llcrypt unregistered [ 668.023937] Key type ._llcrypt registered [ 668.026932] Key type .llcrypt registered [ 668.243613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 668.261489] alg: No test for adler32 (adler32-zlib) [ 669.152807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.226671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 669.255806] LNet: 30754:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 669.263795] LNet: 30754:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 669.272506] LNet: 30754:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 669.286628] LNet: Message forwarding will be enabled in 2s [ 672.351127] LNet: Message forwarding enabled [ 672.720034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 672.748486] LNet: Message forwarding will stop in 2s to 6s [ 675.433192] LNet: Message forwarding disabled [ 676.243367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 676.269148] LNet: 30843:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 676.279115] LNet: 30843:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 676.293275] LNet: 30843:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 676.310478] LNet: Message forwarding will be enabled in 2s [ 676.480195] LNet: 30856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.205422] Key type .llcrypt unregistered [ 677.209596] Key type ._llcrypt unregistered [ 689.577547] Key type ._llcrypt registered [ 689.581139] Key type .llcrypt registered [ 690.802083] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:32:34 (1790699554) [ 691.228265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 691.250798] alg: No test for adler32 (adler32-zlib) [ 692.140421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.238757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 699.331186] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:32:43 (1790699563) [ 699.496533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.526825] LNet: 32272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.561891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.702553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 699.784748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 699.847472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 699.906157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 699.961960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 700.004185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 700.061102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 700.103816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 700.162768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 700.216621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 700.277609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 700.328607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 700.367385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 700.413799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 700.453543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 700.493416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 700.588739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 700.670897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 700.733318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 700.763626] LNetError: 32319:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 706.897196] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:32:51 (1790699571) [ 707.084270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.103509] LNet: 32899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.124810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.227235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.953479] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:32:58 (1790699578) [ 715.690098] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg457-server [ 717.221875] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:33:01 (1790699581) [ 717.345925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.411131] LNet: 33841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.465547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.527496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.576507] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 718.302654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[58-62]@tcp [ 718.327045] LNet: 33976:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 718.378961] LNetError: 33847:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.58@tcp is being used as a gateway but routing feature is not turned on [ 718.403344] LNetError: 33847:0:(lib-lnet.h:1293:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.58@tcp has gone from up to down [ 721.441693] LNetError: 33853:0:(lib-lnet.h:1293:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.62@tcp has gone from up to down [ 721.482803] LNetError: 33853:0:(lib-lnet.h:1293:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 724.481026] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:33:08 (1790699588) [ 726.516726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg457-server [ 727.881856] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:33:12 (1790699592) [ 728.235359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.261331] LNet: 34926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.271894] LNetError: 34926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.299927] LNet: Removed LNI 192.168.204.57@tcp [ 728.324277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.390228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.428229] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 729.088752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[58-63/2]@tcp [ 729.108301] LNet: 35043:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 729.121376] LNet: 35043:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 729.134970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[58-63/2]@tcp [ 729.163944] LNetError: 34930:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.58@tcp is being used as a gateway but routing feature is not turned on [ 729.180424] LNetError: 34930:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 729.191169] LNetError: 34930:0:(lib-lnet.h:1293:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.58@tcp has gone from up to down [ 729.207369] LNetError: 34930:0:(lib-lnet.h:1293:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 735.303493] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:33:19 (1790699599) [ 735.563373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.594399] LNet: 35644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.604044] LNetError: 35644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.619605] LNet: Removed LNI 192.168.204.57@tcp [ 736.649377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.779991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 736.890044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.929464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.976100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 737.024216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.067367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.115902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 737.201527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.245610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.336121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 737.412923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.442669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.479198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 737.541495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.571962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.630429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 737.691269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.720621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.766366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 737.830110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.859834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.912347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 737.961363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.000742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.082107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 738.158198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.211592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.312816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 738.406982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.441630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.500358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 738.571903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.610510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.663434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 744.505866] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:33:28 (1790699608) [ 746.429402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg457-server [ 748.090993] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:33:32 (1790699612) [ 750.025805] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg457-server [ 751.822933] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:33:35 (1790699615) [ 751.924373] LNet: 37057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.935245] LNet: 37057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 752.575808] Key type .llcrypt unregistered [ 752.581198] Key type ._llcrypt unregistered [ 758.570729] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 758.831901] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 758.849028] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 758.851698] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 761.384180] Key type ._llcrypt registered [ 761.389863] Key type .llcrypt registered [ 761.600116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 761.620860] alg: No test for adler32 (adler32-zlib) [ 762.535230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.582326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 762.609784] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 763.262838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 763.289645] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 765.315061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 765.339725] LNet: Removed LNI 10.1.2.3@tcp [ 765.474309] LNet: 37788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.481242] LNetError: 37788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.494933] LNet: Removed LNI 192.168.204.57@tcp [ 766.155390] Key type .llcrypt unregistered [ 766.159877] Key type ._llcrypt unregistered [ 777.732957] Key type ._llcrypt registered [ 777.741460] Key type .llcrypt registered [ 779.009505] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:34:02 (1790699642) [ 779.571509] Key type .llcrypt unregistered [ 779.573298] Key type ._llcrypt unregistered [ 785.677478] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 785.937073] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 785.946061] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 785.949258] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 788.458530] Key type ._llcrypt registered [ 788.460109] Key type .llcrypt registered [ 788.626468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.643029] alg: No test for adler32 (adler32-zlib) [ 789.582650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.642766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.695249] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 790.352806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 790.372631] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.784434] LNet: 39292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.790922] LNetError: 39292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 792.796782] LNet: Removed LNI 10.1.2.3@tcp [ 792.806785] LNet: Removed LNI 192.168.204.57@tcp [ 793.327996] Key type .llcrypt unregistered [ 793.332482] Key type ._llcrypt unregistered [ 806.344514] Key type ._llcrypt registered [ 806.349958] Key type .llcrypt registered [ 808.126955] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:34:31 (1790699671) [ 808.531666] Key type .llcrypt unregistered [ 808.533360] Key type ._llcrypt unregistered [ 815.126427] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 815.384792] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 815.408929] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 815.414147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 818.052060] Key type ._llcrypt registered [ 818.055574] Key type .llcrypt registered [ 818.250102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 818.276612] alg: No test for adler32 (adler32-zlib) [ 819.269776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.340512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 819.370134] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 821.308452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 821.336326] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.713930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 823.751309] LNet: Removed LNI 10.1.2.3@tcp [ 823.772981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 823.813993] LNet: Removed LNI 10.1.2.31@tcp [ 823.841113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 823.882092] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.732582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 824.754391] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 825.704936] LNet: 41223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.713066] LNetError: 41223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 825.723077] LNet: Removed LNI 10.1.2.31@tcp [ 825.736875] LNet: Removed LNI 10.1.2.3@tcp [ 826.457471] Key type .llcrypt unregistered [ 826.459144] Key type ._llcrypt unregistered [ 839.251921] Key type ._llcrypt registered [ 839.255348] Key type .llcrypt registered [ 840.683160] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:35:04 (1790699704) [ 841.269674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 841.307471] alg: No test for adler32 (adler32-zlib) [ 842.259113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.333160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 842.373836] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 843.047830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.068784] LNet: 42185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.076813] LNetError: 42185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.103243] LNet: Removed LNI 192.168.204.57@tcp [ 843.123768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.197802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.57@tcp [ 843.223831] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 844.017951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.57@tcp [ 844.051607] LNet: Removed LNI 192.168.204.57@tcp [ 844.128351] LNet: 42320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.890464] Key type .llcrypt unregistered [ 844.894074] Key type ._llcrypt unregistered [ 856.212785] Key type ._llcrypt registered [ 856.216946] Key type .llcrypt registered [ 857.784539] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:35:21 (1790699721) [ 859.446683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg457-server [ 861.097615] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:35:25 (1790699725) [ 861.524312] Key type .llcrypt unregistered [ 861.526545] Key type ._llcrypt unregistered [ 867.164362] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 867.242830] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 867.248821] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 867.258852] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 869.794563] Key type ._llcrypt registered [ 869.796522] Key type .llcrypt registered [ 869.928388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 869.945350] alg: No test for adler32 (adler32-zlib) [ 870.865086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.918397] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 871.583607] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 873.344404] LNet: 44174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.349662] LNetError: 44174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.353863] LNet: Removed LNI 192.168.204.57@tcp1 [ 873.368933] LNet: Removed LNI 10.1.2.3@tcp2 [ 873.871702] Key type .llcrypt unregistered [ 873.874512] Key type ._llcrypt unregistered [ 885.675967] Key type ._llcrypt registered [ 885.677214] Key type .llcrypt registered [ 887.058405] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:35:51 (1790699751) [ 887.588666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 887.610932] alg: No test for adler32 (adler32-zlib) [ 888.559464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.622401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.650774] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 889.486555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.500925] LNet: 45153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.505562] LNetError: 45153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.521229] LNet: Removed LNI 192.168.204.57@tcp [ 889.545062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.600159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 889.618569] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 898.237691] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:36:01 (1790699761) [ 898.381915] LNet: 45744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.392286] LNetError: 45744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.420107] LNet: Removed LNI 192.168.204.57@tcp [ 899.626743] Key type .llcrypt unregistered [ 899.637828] Key type ._llcrypt unregistered [ 908.678828] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 908.938678] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 908.960853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 908.970309] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 911.690909] Key type ._llcrypt registered [ 911.692601] Key type .llcrypt registered [ 911.995312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.025917] alg: No test for adler32 (adler32-zlib) [ 913.155555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.251699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 913.292281] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 914.567446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 914.625021] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 917.963846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.990945] LNet: 46498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.996062] LNetError: 46498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.001740] LNet: Removed LNI 10.1.2.3@tcp [ 918.019977] LNet: Removed LNI 192.168.204.57@tcp [ 918.046947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.160739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 918.208815] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 918.220214] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.079178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.136059] LNet: 46525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.152990] LNetError: 46525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.167324] LNetError: 46525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 919.180171] LNet: Removed LNI 10.1.2.3@tcp [ 919.224160] LNet: Removed LNI 192.168.204.57@tcp [ 919.271320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.371818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 919.428352] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 919.445893] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.795867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 919.831679] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 921.107904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.157574] LNet: 46688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.167993] LNetError: 46688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.178778] LNetError: 46688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 921.193348] LNet: Removed LNI 10.1.2.3@tcp [ 921.202402] LNet: Removed LNI 192.168.204.57@tcp [ 921.223535] LNet: Removed LNI 192.168.204.57@tcp2 [ 921.291225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.370514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 921.429354] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 921.439632] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.451812] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 932.171293] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:36:35 (1790699795) [ 932.344612] LNet: 47286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.362209] LNetError: 47286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.371507] LNetError: 47286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 932.389416] LNet: Removed LNI 10.1.2.3@tcp [ 932.391019] LNet: Removed LNI 192.168.204.57@tcp [ 932.440412] LNet: Removed LNI 192.168.204.57@tcp2 [ 933.838163] Key type .llcrypt unregistered [ 933.846626] Key type ._llcrypt unregistered [ 943.881779] Key type ._llcrypt registered [ 943.883861] Key type .llcrypt registered [ 944.109993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 944.127710] alg: No test for adler32 (adler32-zlib) [ 945.071322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.241388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 945.273387] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 946.425811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.493661] LNet: 47690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.507271] LNetError: 47690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.533262] LNet: Removed LNI 192.168.204.57@tcp [ 946.567603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.645433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 946.681816] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 947.384203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.429619] LNet: 47721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.454154] LNetError: 47721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.493196] LNet: Removed LNI 192.168.204.57@tcp [ 947.517514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.562632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 947.579585] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 947.904644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.931081] LNet: Removed LNI 192.168.204.57@tcp [ 947.957709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.003519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 948.024993] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 948.383966] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 948.494808] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 948.500982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 948.508393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 949.733436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.759249] LNet: 47820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.765480] LNet: 47820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 949.776743] LNetError: 47820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.784123] LNetError: 47820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 949.808806] LNet: Removed LNI 192.168.204.57@tcp [ 949.864467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.001327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 950.046237] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 957.425622] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:37:01 (1790699821) [ 957.756893] LNet: 48426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.770062] LNetError: 48426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.804131] LNet: Removed LNI 192.168.204.57@tcp [ 958.414895] Key type .llcrypt unregistered [ 958.417689] Key type ._llcrypt unregistered [ 964.268385] Key type ._llcrypt registered [ 964.270335] Key type .llcrypt registered [ 964.416108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.435587] alg: No test for adler32 (adler32-zlib) [ 965.313488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.354520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 965.375408] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 966.100845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.117174] LNet: 48825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.122628] LNetError: 48825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.138257] LNet: Removed LNI 192.168.204.57@tcp [ 966.171286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.253577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 966.294841] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 972.311925] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:37:16 (1790699836) [ 972.576768] LNet: 49425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.583772] LNetError: 49425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.599739] LNet: Removed LNI 192.168.204.57@tcp [ 973.300669] Key type .llcrypt unregistered [ 973.302949] Key type ._llcrypt unregistered [ 978.801537] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 978.956238] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 978.964644] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 978.968891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 981.550851] Key type ._llcrypt registered [ 981.554878] Key type .llcrypt registered [ 981.768900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 981.788853] alg: No test for adler32 (adler32-zlib) [ 982.686112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.789149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 982.833787] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 983.921473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.945668] LNet: 49877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.952325] LNetError: 49877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.966262] LNet: Removed LNI 192.168.204.57@tcp [ 983.993407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.080977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 984.136356] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 991.793617] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:37:35 (1790699855) [ 992.297368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.358284] LNet: 50484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.368708] LNetError: 50484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.397289] LNet: Removed LNI 192.168.204.57@tcp [ 992.433510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.611240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1000.501527] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:37:44 (1790699864) [ 1000.765808] LNet: 51075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.387274] Key type .llcrypt unregistered [ 1001.398328] Key type ._llcrypt unregistered [ 1008.361335] Key type ._llcrypt registered [ 1008.367876] Key type .llcrypt registered [ 1008.646811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1008.689597] alg: No test for adler32 (adler32-zlib) [ 1009.606555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.683140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1009.715327] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1010.554543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.577836] LNet: 51473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.584247] LNetError: 51473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.623398] LNet: Removed LNI 192.168.204.57@tcp [ 1010.656545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.754176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1010.817311] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1018.440379] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:38:02 (1790699882) [ 1018.677927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.720508] LNet: 52074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.742018] LNetError: 52074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.779668] LNet: Removed LNI 192.168.204.57@tcp [ 1018.803433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.882967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1026.833545] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:38:10 (1790699890) [ 1028.738782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1030.555519] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:38:14 (1790699894) [ 1032.137144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1034.105376] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:38:18 (1790699898) [ 1035.799441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1037.895753] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:38:21 (1790699901) [ 1039.783309] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1041.932585] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:38:25 (1790699905) [ 1043.672123] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1045.755459] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:38:29 (1790699909) [ 1045.871640] LNet: 54117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.834916] Key type .llcrypt unregistered [ 1046.836828] Key type ._llcrypt unregistered [ 1056.199475] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1056.555969] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1056.568423] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1056.572420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1059.225523] Key type ._llcrypt registered [ 1059.228690] Key type .llcrypt registered [ 1059.399763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1059.413272] alg: No test for adler32 (adler32-zlib) [ 1060.368080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.473659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1060.507150] LNet: Added LNI 192.168.204.57@tcp [8/512/0/180] [ 1060.518185] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1060.522715] LNet: 54449:0:(api-ni.c:2992:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1060.658294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.676562] LNet: 54468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.683945] LNetError: 54468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1060.693211] LNet: Removed LNI 10.1.2.3@tcp [ 1060.696643] LNetError: 54468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.715206] LNet: Removed LNI 192.168.204.57@tcp [ 1060.732664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.819983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1060.845083] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1060.921652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.939555] LNet: 54492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.946080] LNetError: 54492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.964743] LNet: Removed LNI 192.168.204.57@tcp [ 1060.983644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.047933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1061.080636] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1061.086659] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1061.091360] LNet: 54500:0:(api-ni.c:2992:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1061.189627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.211942] LNet: 54518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.219961] LNetError: 54518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1061.228439] LNet: Removed LNI 10.1.2.3@tcp [ 1061.230785] LNetError: 54518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.243370] LNet: Removed LNI 192.168.204.57@tcp [ 1061.264502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.328617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1061.378366] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1061.387986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1061.393308] LNet: 54526:0:(api-ni.c:2992:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1061.624521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.657973] LNet: 54552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.664805] LNetError: 54552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1061.671138] LNet: Removed LNI 10.1.2.3@tcp [ 1061.674653] LNetError: 54552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.692512] LNet: Removed LNI 192.168.204.57@tcp [ 1061.728766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.780950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1061.814072] LNet: Added LNI 192.168.204.57@tcp [8/256/0/182] [ 1061.820906] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1061.932901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.961293] LNet: 54580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.973551] LNetError: 54580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1061.983465] LNet: Removed LNI 10.1.2.3@tcp [ 1061.989491] LNetError: 54580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.005670] LNet: Removed LNI 192.168.204.57@tcp [ 1062.025862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.117150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1062.148743] LNet: Added LNI 192.168.204.57@tcp [8/256/0/182] [ 1062.153832] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1062.157982] LNet: 54588:0:(api-ni.c:2992:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1062.246510] LNet: Removed LNI 10.1.2.3@tcp [ 1062.262708] LNet: Removed LNI 192.168.204.57@tcp [ 1062.813660] Key type .llcrypt unregistered [ 1062.817898] Key type ._llcrypt unregistered [ 1075.317817] Key type ._llcrypt registered [ 1075.322131] Key type .llcrypt registered [ 1076.686194] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:39:00 (1790699940) [ 1078.370596] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1080.128332] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:39:04 (1790699944) [ 1080.733728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1080.764845] alg: No test for adler32 (adler32-zlib) [ 1081.793892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.913907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1081.955122] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1082.813956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.57 [ 1082.840660] LNet: 55867:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1082.852345] LNet: Removed LNI 192.168.204.57@tcp [ 1089.606963] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:39:13 (1790699953) [ 1089.805436] LNet: 56443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.813377] LNetError: 56443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.842927] LNet: Removed LNI 192.168.204.57@tcp [ 1090.459540] Key type .llcrypt unregistered [ 1090.469620] Key type ._llcrypt unregistered [ 1096.942542] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1097.252922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1098.804130] Key type ._llcrypt registered [ 1098.809273] Key type .llcrypt registered [ 1099.005556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1099.024772] alg: No test for adler32 (adler32-zlib) [ 1100.489602] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1107.430841] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:39:31 (1790699971) [ 1107.714701] LNet: 57361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.729896] LNetError: 57361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1107.750889] LNet: Removed LNI 10.1.2.3@tcp [ 1108.391722] Key type .llcrypt unregistered [ 1108.395717] Key type ._llcrypt unregistered [ 1115.351206] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1115.572587] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1115.582752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1115.586939] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1118.378837] Key type ._llcrypt registered [ 1118.382346] Key type .llcrypt registered [ 1118.605815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1118.641613] alg: No test for adler32 (adler32-zlib) [ 1119.755124] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1126.990974] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:39:51 (1790699991) [ 1128.896612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1130.536187] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:39:54 (1790699994) [ 1130.952586] LNet: 58558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.958182] LNetError: 58558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1130.987841] LNet: Removed LNI 10.1.2.3@tcp [ 1131.638720] Key type .llcrypt unregistered [ 1131.641851] Key type ._llcrypt unregistered [ 1137.764152] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1138.130222] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1138.139674] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1138.149783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1141.240908] Key type ._llcrypt registered [ 1141.245984] Key type .llcrypt registered [ 1141.520643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1141.544242] alg: No test for adler32 (adler32-zlib) [ 1142.470945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.553296] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1153.406664] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:40:17 (1790700017) [ 1154.143367] LNet: 59799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.148302] LNetError: 59799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1154.164046] LNet: Removed LNI 10.1.2.3@tcp [ 1154.957444] Key type .llcrypt unregistered [ 1154.962134] Key type ._llcrypt unregistered [ 1162.687168] Key type ._llcrypt registered [ 1162.693515] Key type .llcrypt registered [ 1163.111476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1163.129276] alg: No test for adler32 (adler32-zlib) [ 1164.156544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1164.214378] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1173.574947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1174.420448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_interrupt -r 1 [ 1174.473482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_interrupt -r 1 [ 1174.527940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1174.557526] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1174.806744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.831721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.859680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.884481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.911866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1174.936947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.960344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.983715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.008076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.025699] LNet: 60241:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.046670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.076837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.112311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.141974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.169952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.201529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.233521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1175.800548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_dropped -r 1 [ 1175.843382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_dropped -r 1 [ 1175.905338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1175.932139] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1176.200874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.242746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.284216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.320969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.353577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1176.393862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.431670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.461747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.492945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.508847] LNet: 60352:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.530888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.544704] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1176.565846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.593397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.627615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.651864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.696821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.771759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1177.424826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_aborted -r 1 [ 1177.465669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_aborted -r 1 [ 1177.536630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1177.581879] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1177.594117] LNet: Skipped 1 previous similar message [ 1177.942496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.997795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.058517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.111520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.191940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1178.267236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.340675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.395693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.456674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.495721] LNet: 60464:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.525953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.576771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.636241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.694614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.753152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.798986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.841578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.347337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_no_route -r 1 [ 1179.393623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_no_route -r 1 [ 1179.436099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1179.623721] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1179.632027] LNet: Skipped 1 previous similar message [ 1179.788973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.816392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.847218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.882585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.934659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1179.976863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.015635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.060600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.103982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.138851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.180973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.213405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.245876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.299834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.369864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.418924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1181.079252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_timeout -r 1 [ 1181.125873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_timeout -r 1 [ 1181.160408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1181.462639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.504672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.538808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.584538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.655875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1181.714994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.775966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.831644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.889902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.927603] LNet: 60686:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.947401] LNet: 60686:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1181.968779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.007931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.051461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.105743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.146064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.195406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.250901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.977618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_error -r 1 [ 1183.023648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_error -r 1 [ 1183.084344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1183.401298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.446063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.482800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.520215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.555492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1183.601265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.649243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.691533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.730423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.771921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.800466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.836156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.874696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.908948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.943066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.978547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1184.340426] LNet: 60839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.349184] LNetError: 60839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1185.401274] LNet: Removed LNI 192.168.204.57@tcp [ 1186.086571] Key type .llcrypt unregistered [ 1186.100831] Key type ._llcrypt unregistered [ 1208.790587] Key type ._llcrypt registered [ 1208.796203] Key type .llcrypt registered [ 1210.469850] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:41:14 (1790700074) [ 1211.665723] Key type .llcrypt unregistered [ 1211.670636] Key type ._llcrypt unregistered [ 1218.341181] Key type ._llcrypt registered [ 1218.343232] Key type .llcrypt registered [ 1218.630740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1218.662526] alg: No test for adler32 (adler32-zlib) [ 1219.550127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1219.591431] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1229.625745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1241.527540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.599148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1241.637065] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1242.484590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1243.254480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_interrupt -r 1 [ 1243.295237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_interrupt -r 1 [ 1243.339232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_interrupt -r 1 [ 1243.378682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_interrupt -r 1 [ 1243.452968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_interrupt -r 1 [ 1243.502311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_interrupt -r 1 [ 1243.542645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_interrupt -r 1 [ 1243.577110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_interrupt -r 1 [ 1243.628430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1243.649965] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1243.657278] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp1 [ 1243.948949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.976208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.004476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.056247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.108445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1244.142717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.186646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.239454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.302814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.339298] LNet: 62337:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.361901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.425247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.460350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.491985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.522655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.566729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.608399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1245.159829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_dropped -r 1 [ 1245.189295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_dropped -r 1 [ 1245.225066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_dropped -r 1 [ 1245.250627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_dropped -r 1 [ 1245.289934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_dropped -r 1 [ 1245.318294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_dropped -r 1 [ 1245.350156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_dropped -r 1 [ 1245.414814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_dropped -r 1 [ 1245.459199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1245.480291] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1245.488390] LNet: Skipped 2 previous similar messages [ 1245.738342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.805326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.858648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.891588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.939345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1246.010784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.061927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.101475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.152536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.176081] LNet: 62460:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1246.222377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.275765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.319321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.342547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.368695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.400301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.437149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1246.982914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_aborted -r 1 [ 1247.015880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_aborted -r 1 [ 1247.070115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_aborted -r 1 [ 1247.141147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_aborted -r 1 [ 1247.187744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_aborted -r 1 [ 1247.252720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_aborted -r 1 [ 1247.316620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_aborted -r 1 [ 1247.366244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_aborted -r 1 [ 1247.399228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1247.413456] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1247.418875] LNet: Skipped 2 previous similar messages [ 1247.663796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.700087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.736742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.783651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.819717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1247.853672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.895963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.939153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.991432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1248.025466] LNet: 62583:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1248.055916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1248.095696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1248.136636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.186911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.238785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.282481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.320238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1248.906679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_no_route -r 1 [ 1248.943380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_no_route -r 1 [ 1248.987825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_no_route -r 1 [ 1249.033243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_no_route -r 1 [ 1249.100930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_no_route -r 1 [ 1249.147134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_no_route -r 1 [ 1249.202794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_no_route -r 1 [ 1249.289753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_no_route -r 1 [ 1249.353912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1249.628991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.664251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1249.714363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.750138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.784419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1249.809370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.865404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.923403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.983366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1250.033493] LNet: 62707:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1250.060231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1250.109816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1250.158558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1250.192722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1250.265425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.342590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.404225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1251.106286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_timeout -r 1 [ 1251.150419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_timeout -r 1 [ 1251.194939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_timeout -r 1 [ 1251.231568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_timeout -r 1 [ 1251.280068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_timeout -r 1 [ 1251.321661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_timeout -r 1 [ 1251.362285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_timeout -r 1 [ 1251.410480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_timeout -r 1 [ 1251.457556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1251.477231] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1251.482461] LNet: Skipped 5 previous similar messages [ 1251.792174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.828316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1251.854864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.890926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.954794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1252.007767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.049893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.099770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.143622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.179553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.209531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.237909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.268027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.300464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1252.343630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.379461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1252.917438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_error -r 1 [ 1252.950168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_error -r 1 [ 1252.989902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_error -r 1 [ 1253.044353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_error -r 1 [ 1253.084358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_error -r 1 [ 1253.131554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_error -r 1 [ 1253.167170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_error -r 1 [ 1253.219623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_error -r 1 [ 1253.269267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1253.531332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1253.579618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1253.619297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1253.666534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1253.709325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1253.748535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1253.776217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1253.841827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1253.899329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1253.935804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1253.970844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.015544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.064310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1254.100290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.153226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1254.189489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.252814] LNet: 63030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.263715] LNetError: 63030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1260.271594] LNet: Removed LNI 192.168.204.57@tcp [ 1260.288151] LNet: Removed LNI 192.168.204.57@tcp1 [ 1260.669608] Key type .llcrypt unregistered [ 1260.675657] Key type ._llcrypt unregistered [ 1278.159933] Key type ._llcrypt registered [ 1278.165715] Key type .llcrypt registered [ 1279.783266] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:42:23 (1790700143) [ 1280.664632] Key type .llcrypt unregistered [ 1280.673331] Key type ._llcrypt unregistered [ 1288.570458] Key type ._llcrypt registered [ 1288.575226] Key type .llcrypt registered [ 1289.042662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1289.092293] alg: No test for adler32 (adler32-zlib) [ 1290.189189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1290.234310] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1300.346954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1301.306900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_dropped -r 1 [ 1301.350762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_dropped -r 1 [ 1301.430376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1301.489323] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1301.871976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1301.912267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1301.945448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1301.980141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1302.012602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1302.044775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.079909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.132597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.174852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.190679] LNet: 64296:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.206779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.239420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.272887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.309394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1302.335416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.374343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.421644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1302.970992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_error -r 1 [ 1303.007774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_error -r 1 [ 1303.049839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1303.066921] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1303.262368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1303.288826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1303.312444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1303.340992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1303.386401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1303.430299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1303.480064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1303.535806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1303.574439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1303.609816] LNet: 64407:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1303.644582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1303.666037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1303.685504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1303.718704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1303.743287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1303.783277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1303.844769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1304.265672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_timeout -r 1 [ 1304.307881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_timeout -r 1 [ 1304.345229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1304.366681] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1304.611762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.652727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1304.687865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.704381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.751112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1304.801547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.837286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.876028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.932488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.978497] LNet: 64518:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.005517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.042793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.084675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.132512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.177596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.233431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.289126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1305.627021] LNet: 64560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.631544] LNetError: 64560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1305.658370] LNet: Removed LNI 192.168.204.57@tcp [ 1306.279481] Key type .llcrypt unregistered [ 1306.288178] Key type ._llcrypt unregistered [ 1325.726277] Key type ._llcrypt registered [ 1325.730384] Key type .llcrypt registered [ 1327.310815] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:43:11 (1790700191) [ 1329.150048] Key type .llcrypt unregistered [ 1329.155729] Key type ._llcrypt unregistered [ 1336.667433] Key type ._llcrypt registered [ 1336.669585] Key type .llcrypt registered [ 1336.946672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1336.960160] alg: No test for adler32 (adler32-zlib) [ 1337.913573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1337.943597] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1346.318466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1356.316200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.384983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1356.436293] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1357.623786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1358.556927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_dropped -r 1 [ 1358.627917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e remote_dropped -r 1 [ 1358.675120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_dropped -r 1 [ 1358.738503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e remote_dropped -r 1 [ 1358.778490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e remote_dropped -r 1 [ 1358.823423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e remote_dropped -r 1 [ 1358.857735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e remote_dropped -r 1 [ 1358.941278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e remote_dropped -r 1 [ 1358.987739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1359.019817] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1359.029792] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.157@tcp1 [ 1359.337891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1359.424284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1359.478217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1359.538242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1359.581140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1359.631487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1359.694563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1359.751908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1359.812916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1359.837548] LNet: 66048:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1359.862554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1359.900770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.938795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.980537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1360.029731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.080848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1360.150862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1360.212688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.997710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_error -r 1 [ 1361.059978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e remote_error -r 1 [ 1361.118207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_error -r 1 [ 1361.201413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e remote_error -r 1 [ 1361.260036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e remote_error -r 1 [ 1361.322929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e remote_error -r 1 [ 1361.371497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e remote_error -r 1 [ 1361.413898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e remote_error -r 1 [ 1361.450189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1361.464885] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1361.471632] LNet: Skipped 2 previous similar messages [ 1361.732615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1361.770780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1361.803938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1361.842829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1361.889387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1361.926908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1361.967437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1362.015682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1362.044992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1362.075660] LNet: 66174:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1362.093689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1362.139914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1362.179586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.215345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.255729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.301784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1362.358929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1362.393657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1363.227470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_timeout -r 1 [ 1363.310716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e remote_timeout -r 1 [ 1363.411917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_timeout -r 1 [ 1363.489433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e remote_timeout -r 1 [ 1363.539589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e remote_timeout -r 1 [ 1363.614975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e remote_timeout -r 1 [ 1363.692343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e remote_timeout -r 1 [ 1363.754638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e remote_timeout -r 1 [ 1363.806935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1363.830630] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1363.837751] LNet: Skipped 1 previous similar message [ 1364.134723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1364.160615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1364.191225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1364.242957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1364.287741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1364.345461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1364.394746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1364.419926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1364.441970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1364.458896] LNet: 66299:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1364.474758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1364.507162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1364.533729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.560658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1364.586082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.614237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1364.646707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1364.675771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1370.762867] LNet: 66377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.776981] LNetError: 66377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.789546] LNet: Removed LNI 192.168.204.57@tcp [ 1370.801791] LNet: Removed LNI 192.168.204.57@tcp1 [ 1371.260492] Key type .llcrypt unregistered [ 1371.265168] Key type ._llcrypt unregistered [ 1388.551670] Key type ._llcrypt registered [ 1388.553823] Key type .llcrypt registered [ 1390.008964] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:44:13 (1790700253) [ 1391.495459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1393.026867] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:44:17 (1790700257) [ 1394.025487] Key type .llcrypt unregistered [ 1394.033972] Key type ._llcrypt unregistered [ 1401.040465] Key type ._llcrypt registered [ 1401.042532] Key type .llcrypt registered [ 1401.219561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1401.235469] alg: No test for adler32 (adler32-zlib) [ 1402.232498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1402.293966] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1410.773796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1411.607653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e network_timeout -r 1 [ 1411.669167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e network_timeout -r 1 [ 1411.720858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1411.745470] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1412.028358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1412.070454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1412.104035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1412.139295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1412.177645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1412.228782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1412.283994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1412.343874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1412.400177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1412.425436] LNet: 67940:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1412.443939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1412.477539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1412.507332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1412.547454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1412.576270] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1412.596476] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1412.599712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1412.643898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1412.961626] LNet: 67980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.981693] LNetError: 67980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1414.063421] LNet: Removed LNI 192.168.204.57@tcp [ 1414.593551] Key type .llcrypt unregistered [ 1414.598379] Key type ._llcrypt unregistered [ 1434.246237] Key type ._llcrypt registered [ 1434.248246] Key type .llcrypt registered [ 1434.397402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1434.413462] alg: No test for adler32 (adler32-zlib) [ 1435.473563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1435.527602] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1445.259942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1456.176249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.239874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1456.273459] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1457.341134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1458.103796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e network_timeout -r 1 [ 1458.136531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e network_timeout -r 1 [ 1458.170911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e network_timeout -r 1 [ 1458.217872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e network_timeout -r 1 [ 1458.273571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e network_timeout -r 1 [ 1458.335287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e network_timeout -r 1 [ 1458.389905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e network_timeout -r 1 [ 1458.436962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e network_timeout -r 1 [ 1458.501360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1458.554893] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1458.807591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1458.837464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1458.873728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1458.922505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1458.974514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1459.004703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1459.060573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1459.092676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1459.105071] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp1 [ 1459.112046] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1459.115165] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.157@tcp1 [ 1459.130574] LNet: Skipped 1 previous similar message [ 1459.148299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1459.169286] LNet: 68874:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1459.195133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1459.228598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1459.280553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1459.324369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1459.366016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1459.405420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1464.322148] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1464.804209] LNet: 68948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.816858] LNetError: 68948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.827509] LNet: Removed LNI 192.168.204.57@tcp [ 1464.855946] LNet: Removed LNI 192.168.204.57@tcp1 [ 1465.359826] Key type .llcrypt unregistered [ 1465.363335] Key type ._llcrypt unregistered [ 1483.588957] Key type ._llcrypt registered [ 1483.596884] Key type .llcrypt registered [ 1485.014300] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:45:48 (1790700348) [ 1485.473350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1485.500385] alg: No test for adler32 (adler32-zlib) [ 1486.430289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.516518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1486.564446] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1487.521704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1487.564662] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1488.625987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 1488.702978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1488.747260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1488.817198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1488.875309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1488.921291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1488.970813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.57@tcp [ 1489.008401] LNetError: 70057:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.57@tcp to recovery [ 1489.567348] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1489.567614] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1491.615492] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1492.607945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1495.713611] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1496.249700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1496.287472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1496.364306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1496.398419] LNet: 70165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.405911] LNetError: 70165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.415803] LNet: Removed LNI 192.168.204.57@tcp [ 1496.432715] LNet: Removed LNI 192.168.204.57@tcp1 [ 1496.452258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.518257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1496.569909] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1497.454945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1497.485789] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1498.343424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 1498.387620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1498.471478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1498.544532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1498.606785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1498.643940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1498.675957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1498.724732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.57@tcp [ 1498.770240] LNetError: 70437:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.57@tcp to recovery [ 1499.552078] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1502.451847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1505.695402] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1505.711892] LNet: Skipped 1 previous similar message [ 1510.681381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1510.760259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1510.848323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1519.031968] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:46:22 (1790700382) [ 1519.449699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.482115] LNet: 71165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.497231] LNetError: 71165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.505113] LNetError: 71165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1519.518819] LNet: Removed LNI 192.168.204.57@tcp [ 1519.530685] LNet: Removed LNI 192.168.204.57@tcp1 [ 1519.550394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.606345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.656322] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1520.693960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1520.730393] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1522.284848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 1522.403350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1522.436836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1522.470033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1522.518377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1522.573981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1522.625448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.57@tcp --health 0 [ 1522.734427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1523.679988] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.57@tcp [ 1523.680533] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1523.701101] LNet: Skipped 1 previous similar message [ 1538.801208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1539.132126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1539.241697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1539.323676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.57@tcp --health 500 [ 1539.569712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1545.368544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1545.482629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.512985] LNet: 71650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.535748] LNetError: 71650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.553715] LNetError: 71650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1545.582055] LNet: Removed LNI 192.168.204.57@tcp [ 1545.603826] LNet: Removed LNI 192.168.204.57@tcp1 [ 1545.654769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.764108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.808975] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1546.691203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1546.710294] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1547.634515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 1547.682055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1547.722192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1547.761523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1547.797731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1547.832691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1547.867487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1547.930574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.57@tcp --health 0 [ 1548.767725] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1548.773685] LNet: Skipped 3 previous similar messages [ 1549.629866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1560.310930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1560.371331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1560.453167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1568.630483] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:47:12 (1790700432) [ 1569.322714] LNet: 72707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.330406] LNetError: 72707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.334637] LNetError: 72707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1569.339527] LNet: Removed LNI 192.168.204.57@tcp [ 1569.355870] LNet: Removed LNI 192.168.204.57@tcp1 [ 1570.198857] Key type .llcrypt unregistered [ 1570.205658] Key type ._llcrypt unregistered [ 1578.490888] Key type ._llcrypt registered [ 1578.497973] Key type .llcrypt registered [ 1578.733742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1578.754621] alg: No test for adler32 (adler32-zlib) [ 1579.893345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1579.934653] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1591.100323] Lustre: DEBUG MARKER: Initial discovery [ 1591.163429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.157@tcp [ 1594.854366] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1594.965273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1595.064393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.157@tcp [ 1595.104326] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1595.165451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1595.230169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.157@tcp -r 1 -m GET -l 3 [ 1595.305663] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1595.354742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.157@tcp -r 1 -m GET -e local_error [ 1595.414422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.157@tcp -r 1 -m PUT -l 6 [ 1597.230364] Lustre: DEBUG MARKER: Force oleg457-client.virtnet to discover 192.168.204.157@tcp (in background) [ 1597.478361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.157@tcp [ 1599.213630] Lustre: DEBUG MARKER: Force oleg457-server to discover 192.168.204.57@tcp [ 1600.543391] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1601.908794] Lustre: DEBUG MARKER: Wait for 73474 [ 1603.791887] Lustre: DEBUG MARKER: Finished wait on 73474 [ 1604.063903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1604.089705] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1604.098716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1604.500481] LNet: 73933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.516710] LNetError: 73933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1604.561240] LNet: Removed LNI 192.168.204.57@tcp [ 1605.063568] Key type .llcrypt unregistered [ 1605.068932] Key type ._llcrypt unregistered [ 1626.954997] Key type ._llcrypt registered [ 1626.959175] Key type .llcrypt registered [ 1628.701737] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:48:12 (1790700492) [ 1629.312600] Key type .llcrypt unregistered [ 1629.317070] Key type ._llcrypt unregistered [ 1637.060649] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1637.303160] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1637.317380] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1637.326802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1639.065819] Key type ._llcrypt registered [ 1639.072768] Key type .llcrypt registered [ 1639.395384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1639.416876] alg: No test for adler32 (adler32-zlib) [ 1640.434411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.525238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.559965] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1641.865855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1641.925940] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1652.314667] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:48:36 (1790700516) [ 1652.618679] LNet: 76023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.627599] LNetError: 76023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1652.641292] LNet: Removed LNI 10.1.2.3@tcp [ 1652.660147] LNet: Removed LNI 192.168.204.57@tcp [ 1653.494576] Key type .llcrypt unregistered [ 1653.502581] Key type ._llcrypt unregistered [ 1662.841204] Key type ._llcrypt registered [ 1662.844619] Key type .llcrypt registered [ 1663.125502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1663.159652] alg: No test for adler32 (adler32-zlib) [ 1664.125315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.219211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1664.240564] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1665.138620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1665.161836] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1666.688914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.57@tcp [ 1667.807375] LNetError: Unexpected error -22 connecting to 192.168.204.57@tcp at host 192.168.204.57:988 [ 1676.457901] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:48:59 (1790700539) [ 1676.932800] LNet: 77165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.946483] LNetError: 77165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1676.955972] LNet: Removed LNI 10.1.2.3@tcp [ 1676.971263] LNet: Removed LNI 192.168.204.57@tcp [ 1677.718530] Key type .llcrypt unregistered [ 1677.723887] Key type ._llcrypt unregistered [ 1687.317493] Key type ._llcrypt registered [ 1687.320467] Key type .llcrypt registered [ 1687.576660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1687.598834] alg: No test for adler32 (adler32-zlib) [ 1688.502530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.580061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1688.619895] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1689.803499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1689.840398] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 1690.593752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.57@tcp1 --nid 192.168.204.57@tcp2 [ 1695.464407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp1 192.168.204.57@tcp2 [ 1695.513447] LNetError: 78002:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.57@tcp1 to 192.168.204.57@tcp2 - Route table may be misconfigured [ 1695.557041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp2 192.168.204.57@tcp1 [ 1703.661385] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:49:27 (1790700567) [ 1703.809813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.858553] LNet: 78577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.874794] LNetError: 78577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.899598] LNet: Removed LNI 192.168.204.57@tcp1 [ 1704.939610] LNet: Removed LNI 192.168.204.57@tcp2 [ 1705.011327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.139753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.169359] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1706.040604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1706.078866] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1707.364609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 1707.430083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.57@tcp [ 1707.485330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp1 192.168.204.57@tcp1 [ 1707.539977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.57@tcp -d 192.168.204.57@tcp -e network_timeout [ 1707.583410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -e network_timeout [ 1707.664790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -e network_timeout [ 1707.712580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -e network_timeout [ 1707.766322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1707.810770] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1707.838050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1707.960363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1708.099081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1708.129761] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1716.677273] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:49:40 (1790700580) [ 1716.763468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.779660] LNet: 79447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.785659] LNetError: 79447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.791262] LNetError: 79447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1716.796765] LNet: Removed LNI 192.168.204.57@tcp [ 1717.862541] LNet: Removed LNI 192.168.204.57@tcp1 [ 1717.906683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.095395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1718.421754] LNet: 79490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.963744] Key type .llcrypt unregistered [ 1718.973534] Key type ._llcrypt unregistered [ 1735.358640] Key type ._llcrypt registered [ 1735.360803] Key type .llcrypt registered [ 1736.919260] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:50:00 (1790700600) [ 1737.462100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1737.482417] alg: No test for adler32 (adler32-zlib) [ 1738.373433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.054381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1742.747942] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:50:06 (1790700606) [ 1742.969740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.027403] LNet: 80596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.148823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.386920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1743.443446] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1744.698561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1744.742917] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1746.552241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1746.650800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp1 [ 1746.789943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp1 [ 1756.450640] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:50:20 (1790700620) [ 1758.900993] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg457-server [ 1761.217781] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:50:25 (1790700625) [ 1763.382130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg457-server [ 1765.781381] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:50:29 (1790700629) [ 1767.788317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg457-server [ 1770.014836] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:50:33 (1790700633) [ 1772.215064] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg457-server [ 1774.918886] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:50:38 (1790700638) [ 1777.162192] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg457-server [ 1779.428691] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:50:43 (1790700643) [ 1782.436611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg457-server [ 1785.224851] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:50:48 (1790700648) [ 1787.482635] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg457-server [ 1789.248225] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:50:53 (1790700653) [ 1791.759427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg457-server [ 1793.559691] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:50:57 (1790700657) [ 1796.154878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg457-server [ 1798.362801] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:51:02 (1790700662) [ 1798.739273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.782572] LNet: 84544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.793957] LNetError: 84544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.805645] LNet: Removed LNI 192.168.204.57@tcp [ 1799.846813] LNet: Removed LNI 192.168.204.57@tcp1 [ 1799.893185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.966179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1800.024321] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1800.748983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1800.875421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1801.160813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.203593] LNet: 84691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.225953] LNetError: 84691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.241903] LNetError: 84691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1801.264375] LNet: Removed LNI 192.168.204.57@tcp [ 1801.319692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.368904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1801.390643] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1802.039612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1802.229336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.291945] LNetError: 84836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.314215] LNet: Removed LNI 192.168.204.57@tcp [ 1802.351680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.424687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.445880] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1803.393152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1803.532314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1803.832675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.863292] LNet: 84984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.869601] LNet: 84984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1804.903624] LNet: Removed LNI 192.168.204.57@tcp [ 1804.961424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.074984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1805.099987] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1806.231143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1806.424236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.445489] LNetError: 85129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.450346] LNetError: 85129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1807.462902] LNet: Removed LNI 192.168.204.57@tcp [ 1807.485826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.568974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1807.597635] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1808.350350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1808.482719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1808.757991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.775550] LNet: 85276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.780524] LNet: 85276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1809.832429] LNet: Removed LNI 192.168.204.57@tcp [ 1809.866589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.928387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1809.964099] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1810.810210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1811.168596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.203770] LNetError: 85422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.216853] LNetError: 85422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1811.245913] LNet: Removed LNI 192.168.204.57@tcp [ 1811.273363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.343710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1811.395838] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1812.357974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1812.432706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1812.629888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.675583] LNet: Removed LNI 192.168.204.57@tcp [ 1813.718995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.808854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1813.853572] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1814.735778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1815.069257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.175743] LNet: Removed LNI 192.168.204.57@tcp [ 1816.233787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.350781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1816.387857] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1817.239623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1817.415496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1817.940747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.989285] LNet: 85862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.998113] LNet: 85862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1819.050956] LNet: Removed LNI 192.168.204.57@tcp [ 1819.103779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.196083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1819.228291] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1820.206167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1820.534377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.592063] LNetError: 86007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.600168] LNetError: 86007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1820.620479] LNet: Removed LNI 192.168.204.57@tcp [ 1820.648461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.693312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1820.748460] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1821.856919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1821.935273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1822.635960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.720466] LNet: Removed LNI 192.168.204.57@tcp [ 1823.808222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.895881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1823.925830] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1825.048487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1825.446777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.534397] LNet: Removed LNI 192.168.204.57@tcp [ 1826.547501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.617077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1826.674141] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1827.409186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1827.473856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.504729] LNet: Removed LNI 192.168.204.57@tcp [ 1827.532734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.595720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.631856] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1828.514148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1828.641812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.685636] LNet: Removed LNI 192.168.204.57@tcp [ 1828.716714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.776764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1828.883996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.936969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.011376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1829.108200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.158487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.202856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1829.332604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.375698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.416183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1829.434706] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1830.321372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.359307] LNet: Removed LNI 192.168.204.57@tcp [ 1830.407777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.513510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1842.209964] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:51:44 (1790700704) [ 1842.348713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1842.401865] LNet: 87324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.420740] LNet: 87324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1842.499576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1842.735188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1842.853117] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/180] [ 1845.030407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.079402] LNetError: 87470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.084707] LNetError: 87470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1845.106478] LNet: Removed LNI 192.168.204.57@tcp231 [ 1845.127128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.199503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1845.230873] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/180] [ 1845.242398] LNet: 87478:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1846.236417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1846.291076] LNet: Removed LNI 192.168.204.57@tcp231 [ 1846.334759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1846.396744] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/180] [ 1847.946822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1848.014243] LNet: Removed LNI 192.168.204.57@tcp231 [ 1848.038945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.104832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1848.134755] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/0] [ 1848.138350] LNet: 87631:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1848.508544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1848.532796] LNet: Removed LNI 192.168.204.57@tcp231 [ 1848.551372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1848.571755] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/0] [ 1849.571677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.621974] LNet: Removed LNI 192.168.204.57@tcp231 [ 1849.652808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1849.745968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1849.769978] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/60] [ 1849.774287] LNet: 87782:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1850.253100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1850.315187] LNet: Removed LNI 192.168.204.57@tcp231 [ 1850.396600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1850.438854] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/60] [ 1851.524983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.579462] LNet: Removed LNI 192.168.204.57@tcp231 [ 1851.617817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.708814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1851.756069] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/180] [ 1852.352173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1852.410612] LNet: Removed LNI 192.168.204.57@tcp231 [ 1852.438238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1852.480960] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/180] [ 1862.657522] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:52:06 (1790700726) [ 1862.839410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1862.918842] LNet: Removed LNI 192.168.204.57@tcp231 [ 1862.943702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.012109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1863.051602] LNet: Added LNI 192.168.204.57@tcp232 [8/256/0/180] [ 1863.907122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1863.956098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1864.012792] LNet: Removed LNI 192.168.204.57@tcp232 [ 1864.051412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1864.113160] LNet: Added LNI 192.168.204.57@tcp232 [8/256/0/180] [ 1865.300317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1865.351327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1865.391224] LNet: Removed LNI 192.168.204.57@tcp232 [ 1865.428848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1865.493314] LNet: Added LNI 192.168.204.57@tcp232 [8/256/0/180] [ 1866.313375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1866.417666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1866.505630] LNet: Removed LNI 192.168.204.57@tcp232 [ 1866.536359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1866.605486] LNet: Added LNI 192.168.204.57@tcp232 [8/256/0/180] [ 1867.891371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1867.949609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1867.986337] LNet: Removed LNI 192.168.204.57@tcp232 [ 1868.018043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1876.169679] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:52:19 (1790700739) [ 1877.196707] LNet: 89791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.204100] LNet: 89791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1878.157620] Key type .llcrypt unregistered [ 1878.170246] Key type ._llcrypt unregistered [ 1885.666639] Key type ._llcrypt registered [ 1885.675900] Key type .llcrypt registered [ 1885.998336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1886.039600] alg: No test for adler32 (adler32-zlib) [ 1887.045194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1887.087139] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1897.031286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1909.968829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.024572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1910.046581] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1911.449599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1912.572416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_interrupt -r 2 [ 1912.630774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_interrupt -r 2 [ 1912.684164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_interrupt -r 2 [ 1912.760935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_interrupt -r 2 [ 1912.811598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_interrupt -r 2 [ 1912.902913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_interrupt -r 2 [ 1912.965804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_interrupt -r 2 [ 1913.023966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_interrupt -r 2 [ 1913.078747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1913.097721] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1913.103502] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1913.580493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1913.628748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1913.690765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1913.721695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1913.748291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1913.782398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1913.847491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1913.897577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1913.939022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1913.961279] LNet: 90462:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1913.976323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1914.000311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1914.033794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1914.068711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1914.100634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1914.141047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1915.070861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_dropped -r 2 [ 1915.184404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_dropped -r 2 [ 1915.294094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_dropped -r 2 [ 1915.398576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_dropped -r 2 [ 1915.484697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_dropped -r 2 [ 1915.590183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_dropped -r 2 [ 1915.687302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_dropped -r 2 [ 1915.789693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_dropped -r 2 [ 1915.855510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1915.897689] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1916.391221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1916.445603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1916.476903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1916.508781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1916.541983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1916.575260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1916.622610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1916.679963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1916.723244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1916.745872] LNet: 90585:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1916.769756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1916.812925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1916.918599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1916.982205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1917.047692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1917.128938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1917.871684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_aborted -r 2 [ 1917.942859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_aborted -r 2 [ 1917.999728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_aborted -r 2 [ 1918.093689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_aborted -r 2 [ 1918.149492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_aborted -r 2 [ 1918.226230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_aborted -r 2 [ 1918.287804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_aborted -r 2 [ 1918.389522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_aborted -r 2 [ 1918.454989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1918.478532] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1918.908080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1918.958309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1919.012182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1919.052282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1919.098526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1919.129888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1919.176274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1919.229262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1919.269541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1919.313564] LNet: 90707:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1919.339483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1919.388749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1919.441798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1919.481645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1919.511712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1919.545581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1920.387950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_no_route -r 2 [ 1920.504614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_no_route -r 2 [ 1920.554336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_no_route -r 2 [ 1920.621718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_no_route -r 2 [ 1920.680795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_no_route -r 2 [ 1920.740463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_no_route -r 2 [ 1920.780347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_no_route -r 2 [ 1920.821713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_no_route -r 2 [ 1920.890172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1920.944707] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1921.387390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1921.430847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1921.469839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1921.507370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1921.549770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1921.599114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1921.685068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1921.747894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1921.812896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1921.851437] LNet: 90829:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1921.877114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1921.924545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1921.955296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1921.986776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1922.027373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1922.067209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1922.811780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_timeout -r 2 [ 1922.889492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_timeout -r 2 [ 1922.942781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_timeout -r 2 [ 1922.985162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_timeout -r 2 [ 1923.041554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_timeout -r 2 [ 1923.107871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_timeout -r 2 [ 1923.161991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_timeout -r 2 [ 1923.198968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_timeout -r 2 [ 1923.248787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1923.587486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1923.630798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1923.667719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1923.714801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1923.768748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1923.821079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1923.887653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1923.938530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1924.004088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1924.080265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1924.125086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1924.175923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1924.217916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1924.269283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1924.309279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1925.004420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_dropped -r 2 [ 1925.053529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e remote_dropped -r 2 [ 1925.102851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_dropped -r 2 [ 1925.132529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e remote_dropped -r 2 [ 1925.177459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e remote_dropped -r 2 [ 1925.229738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e remote_dropped -r 2 [ 1925.283291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e remote_dropped -r 2 [ 1925.335855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e remote_dropped -r 2 [ 1925.390195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1925.418884] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1925.435688] LNet: Skipped 1 previous similar message [ 1925.443301] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.157@tcp [ 1925.768480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1925.825176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1925.862180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1925.888930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1925.944831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1925.975280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1926.003372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1926.035812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1926.065859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1926.089395] LNet: 91073:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1926.101329] LNet: 91073:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1926.138934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1926.164372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1926.195822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1926.238176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1926.273872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1926.308116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1933.430577] LNet: 91148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.444649] LNetError: 91148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.474678] LNet: Removed LNI 192.168.204.57@tcp [ 1933.496360] LNet: Removed LNI 192.168.204.57@tcp1 [ 1934.036966] Key type .llcrypt unregistered [ 1934.044306] Key type ._llcrypt unregistered [ 1953.964660] Key type ._llcrypt registered [ 1953.970534] Key type .llcrypt registered [ 1955.226628] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:53:39 (1790700819) [ 1955.819533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.827460] alg: No test for adler32 (adler32-zlib) [ 1956.693951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.760770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1956.779149] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1957.555435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1957.575579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.57@tcp [ 1957.600207] LNetError: 92136:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.57@tcp to recovery [ 1957.710690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.57@tcp [ 1957.821778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1964.647100] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:53:48 (1790700828) [ 1964.859110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.885631] LNet: 92734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.896431] LNetError: 92734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.918653] LNet: Removed LNI 192.168.204.57@tcp [ 1964.954595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.005760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.026858] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1965.845989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1965.876556] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1967.036288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.57@tcp [ 1967.063579] LNetError: 93002:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.57@tcp to recovery [ 1967.093351] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1967.100447] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1967.305844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.57@tcp [ 1980.411420] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:54:04 (1790700844) [ 1983.308228] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg457-server [ 1986.106942] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:54:09 (1790700849) [ 1988.549922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg457-server [ 1990.662397] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:54:14 (1790700854) [ 1992.626956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg457-server [ 1994.798840] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:54:18 (1790700858) [ 1994.972953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.025584] LNet: 94651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.041357] LNetError: 94651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.063187] LNet: Removed LNI 192.168.204.57@tcp [ 1996.065382] LNetError: 94651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.087185] LNet: Removed LNI 192.168.204.57@tcp1 [ 1996.155892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.277623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1996.335203] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1997.222296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1997.335602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1997.414797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1997.515640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1997.575086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1997.661683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2004.817815] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:54:29 (1790700869) [ 2005.020477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.046465] LNet: 95386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.054522] LNetError: 95386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.068673] LNet: Removed LNI 192.168.204.57@tcp [ 2005.141105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.202771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2005.221687] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2013.155939] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:54:37 (1790700877) [ 2013.368593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2013.387807] LNet: 96098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.396791] LNetError: 96098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.425624] LNet: Removed LNI 192.168.204.57@tcp [ 2013.457695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.556081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.587258] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2023.455421] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:54:47 (1790700887) [ 2025.358760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2028.158475] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:54:51 (1790700891) [ 2028.856964] LNet: 97152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.862833] LNetError: 97152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.881680] LNet: Removed LNI 192.168.204.57@tcp [ 2029.739719] Key type .llcrypt unregistered [ 2029.745842] Key type ._llcrypt unregistered [ 2038.228784] Key type ._llcrypt registered [ 2038.236841] Key type .llcrypt registered [ 2038.441307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2038.461042] alg: No test for adler32 (adler32-zlib) [ 2039.459770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2039.503226] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2046.977379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 2051.720470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.157@tcp [ 2052.036760] LNet: 97570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.046644] LNetError: 97570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.069641] LNet: Removed LNI 192.168.204.57@tcp [ 2052.503996] Key type .llcrypt unregistered [ 2052.513282] Key type ._llcrypt unregistered [ 2063.988351] Key type ._llcrypt registered [ 2063.990502] Key type .llcrypt registered [ 2065.414215] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2067.275496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2069.149228] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:55:33 (1790700933) [ 2071.066492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2072.785385] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:55:37 (1790700937) [ 2074.400239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg457-server [ 2076.523884] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:55:40 (1790700940) [ 2078.493587] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg457-server [ 2080.538557] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:55:44 (1790700944) [ 2081.884658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.048775] alg: No test for adler32 (adler32-zlib) [ 2083.217115] Lustre: Lustre: Build Version: 2.17.59_2_ga235f42 [ 2083.574902] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2085.343753] Key type lgssc registered [ 2086.829803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2111.022866] Key type lgssc unregistered [ 2111.386898] LNet: 100821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.396265] LNetError: 100821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2111.428894] LNet: Removed LNI 192.168.204.57@tcp [ 2112.279134] Key type .llcrypt unregistered [ 2112.281402] Key type ._llcrypt unregistered [ 2130.138377] Key type ._llcrypt registered [ 2130.141874] Key type .llcrypt registered [ 2131.625216] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:56:35 (1790700995) [ 2132.107752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2132.126849] alg: No test for adler32 (adler32-zlib) [ 2133.089630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.179289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2133.229687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2133.276921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2133.317770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2140.930106] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:56:45 (1790701005) [ 2141.032616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.082599] LNet: 102256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.151623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.241969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2141.318156] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2142.446759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.491154] LNet: 102407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.496920] LNetError: 102407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2142.514724] LNet: Removed LNI 192.168.204.57@tcp [ 2142.564468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.613966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2142.646742] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2143.284248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.352513] LNetError: 102435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2143.384959] LNet: Removed LNI 192.168.204.57@tcp [ 2143.433326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.530768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2143.576842] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2151.941540] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:56:55 (1790701015) [ 2152.090728] LNet: 103031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.104473] LNet: 103031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2152.109812] LNetError: 103031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.133700] LNet: Removed LNI 192.168.204.57@tcp [ 2152.927566] Key type .llcrypt unregistered [ 2152.933881] Key type ._llcrypt unregistered [ 2159.769435] Key type ._llcrypt registered [ 2159.771666] Key type .llcrypt registered [ 2160.042877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2160.068221] alg: No test for adler32 (adler32-zlib) [ 2160.987093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.072315] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2161.076269] LNet: *** cfs_fail_loc=e003, val=100*** [ 2161.894357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 2161.925536] LNet: *** cfs_fail_loc=e003, val=100*** [ 2161.931111] LNet: Skipped 2 previous similar messages [ 2161.940078] Ping reply received with mapping: [ 2161.941793] found metadata reply NI[192.168.204.57@tcp] [ 2161.945897] 39 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2161.949360] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2161.952853] 5a 5a 5a 5a 5a 5a 5a 5a [ 2162.068731] LNet: 103432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2162.074724] LNetError: 103432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2163.120560] LNet: Removed LNI 192.168.204.57@tcp [ 2163.791055] Key type .llcrypt unregistered [ 2163.792839] Key type ._llcrypt unregistered [ 2176.342702] Key type ._llcrypt registered [ 2176.346993] Key type .llcrypt registered [ 2177.989492] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:57:21 (1790701041) [ 2178.625115] Key type .llcrypt unregistered [ 2178.633965] Key type ._llcrypt unregistered [ 2186.242095] Key type ._llcrypt registered [ 2186.250025] Key type .llcrypt registered [ 2186.509735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2186.533229] alg: No test for adler32 (adler32-zlib) [ 2187.730611] LNetError: 104487:0:(api-ni.c:2856:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2188.137485] Key type .llcrypt unregistered [ 2188.143227] Key type ._llcrypt unregistered [ 2195.803934] Key type ._llcrypt registered [ 2195.805542] Key type .llcrypt registered [ 2197.563786] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:57:41 (1790701061) [ 2198.151358] Key type .llcrypt unregistered [ 2198.153360] Key type ._llcrypt unregistered [ 2206.456332] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2206.469405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2209.401648] Key type ._llcrypt registered [ 2209.410688] Key type .llcrypt registered [ 2209.917717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2209.961290] alg: No test for adler32 (adler32-zlib) [ 2211.134344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.280862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2223.695457] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:58:07 (1790701087) [ 2223.932804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2223.963518] LNet: 106284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.968904] LNetError: 106284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2223.978828] LNet: Removed LNI 10.1.2.3@tcp [ 2224.001892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2224.772339] Key type .llcrypt unregistered [ 2224.777773] Key type ._llcrypt unregistered [ 2232.023921] Key type ._llcrypt registered [ 2232.027652] Key type .llcrypt registered [ 2232.356128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2232.414289] alg: No test for adler32 (adler32-zlib) [ 2233.397511] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2233.498679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 2233.727085] LNet: 106574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.754214] LNetError: 106574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.798076] LNet: Removed LNI 192.168.204.57@tcp [ 2236.013981] Key type .llcrypt unregistered [ 2236.021764] Key type ._llcrypt unregistered [ 2251.037422] Key type ._llcrypt registered [ 2251.042761] Key type .llcrypt registered [ 2252.707379] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:58:36 (1790701116) [ 2253.341717] Key type .llcrypt unregistered [ 2253.346093] Key type ._llcrypt unregistered [ 2260.102377] Key type ._llcrypt registered [ 2260.105641] Key type .llcrypt registered [ 2260.381426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2260.404810] alg: No test for adler32 (adler32-zlib) [ 2268.948069] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:58:53 (1790701133) [ 2269.358462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2269.415609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.511422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2269.536875] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2270.721927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2270.807505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2270.855030] LNet: Removed LNI 192.168.204.57@tcp [ 2271.348815] LNet: 108339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.193202] Key type .llcrypt unregistered [ 2272.200776] Key type ._llcrypt unregistered [ 2286.949903] Key type ._llcrypt registered [ 2286.957218] Key type .llcrypt registered [ 2288.635214] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:59:12 (1790701152) [ 2289.400744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2289.423212] alg: No test for adler32 (adler32-zlib) [ 2290.410367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.544824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2290.602553] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2292.180647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 2292.283626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.57@tcp [ 2292.332108] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2292.336099] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2292.341575] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2292, 0, 0 [ 2292.346566] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2292.348611] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2292.351162] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2292.353672] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2292, 0, 0 [ 2292.357250] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2292.359602] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2292.361551] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2292.364219] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2292, 0, 2342 [ 2292.367952] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2292.370689] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2292.372725] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2292.374719] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2292, 0, 2342 [ 2292.377320] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2299.810858] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:59:23 (1790701163) [ 2300.680561] LNet: 109902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.687115] LNetError: 109902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.734829] LNet: Removed LNI 192.168.204.57@tcp [ 2302.440624] Key type .llcrypt unregistered [ 2302.443796] Key type ._llcrypt unregistered [ 2309.553629] Key type ._llcrypt registered [ 2309.555663] Key type .llcrypt registered [ 2309.732163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2309.754562] alg: No test for adler32 (adler32-zlib) [ 2310.743441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2310.775703] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2318.531519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 2328.630143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.713295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2328.765593] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 2329.619383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 2329.899952] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2330.080318] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2330.085862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2330.093481] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2332.402308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2332.424945] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2343.563820] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:00:07 (1790701207) [ 2344.240322] LNet: 111422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.258114] LNetError: 111422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.275368] LNet: Removed LNI 192.168.204.57@tcp [ 2345.311185] LNet: Removed LNI 192.168.204.57@tcp1 [ 2345.315914] LNetError: 111422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2345.323030] LNetError: 111422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2345.340693] LNet: Removed LNI 10.1.2.3@tcp99 [ 2345.800179] Key type .llcrypt unregistered [ 2345.802700] Key type ._llcrypt unregistered [ 2354.910705] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2354.914680] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2357.557387] Key type ._llcrypt registered [ 2357.559611] Key type .llcrypt registered [ 2357.742088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2357.760424] alg: No test for adler32 (adler32-zlib) [ 2358.754676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.837972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2358.869805] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2359.580494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2359.596846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2361.753747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2361.811159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2361.850829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2361.868844] LNetError: 112169:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2361.917683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2361.952545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2362.403148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2362.523447] LNet: 112189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.554376] LNetError: 112189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2362.569993] LNet: Removed LNI 10.1.2.3@tcp [ 2363.625661] LNet: Removed LNI 192.168.204.57@tcp [ 2364.603430] Key type .llcrypt unregistered [ 2364.607209] Key type ._llcrypt unregistered [ 2379.572909] Key type ._llcrypt registered [ 2379.577059] Key type .llcrypt registered [ 2381.256686] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:00:44 (1790701244) [ 2381.869530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2381.892503] alg: No test for adler32 (adler32-zlib) [ 2382.854562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.922088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2382.955329] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2384.038023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 2384.157819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg457-client@tcp [ 2391.790865] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:00:55 (1790701255) [ 2392.268761] LNet: 113740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.287735] LNetError: 113740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2393.325276] LNet: Removed LNI 192.168.204.57@tcp [ 2394.220444] Key type .llcrypt unregistered [ 2394.223426] Key type ._llcrypt unregistered [ 2402.148096] Key type ._llcrypt registered [ 2402.150239] Key type .llcrypt registered [ 2402.328393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2402.355644] alg: No test for adler32 (adler32-zlib) [ 2403.250544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2403.276542] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2403.301870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2411.359073] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:01:15 (1790701275) [ 2413.400750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2415.360380] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:01:19 (1790701279) [ 2417.333754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2419.617461] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:01:23 (1790701283) [ 2420.199624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2420.231891] LNet: 115209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.241291] LNetError: 115209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.254229] LNet: Removed LNI 192.168.204.57@tcp [ 2420.277594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2420.412031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2420.462860] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2421.263722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2421.305460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2430.100476] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:01:33 (1790701293) [ 2430.281481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2430.316875] LNet: 115922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2430.329633] LNetError: 115922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2430.356771] LNet: Removed LNI 192.168.204.57@tcp [ 2430.409979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2430.511302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2430.551625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2431.632180] Key type .llcrypt unregistered [ 2431.634816] Key type ._llcrypt unregistered [ 2445.255699] Key type ._llcrypt registered [ 2445.262103] Key type .llcrypt registered [ 2446.916276] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:01:50 (1790701310) [ 2447.390677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.414225] alg: No test for adler32 (adler32-zlib) [ 2448.357933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2448.407488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2448.430372] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2449.217434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2449.284197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2456.934056] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:02:00 (1790701320) [ 2457.141536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2457.174476] LNet: 117499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.187891] LNetError: 117499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.216861] LNet: Removed LNI 192.168.204.57@tcp [ 2457.252979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2457.306952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2457.348880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2464.668335] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 13:02:08 (1790701328) [ 2464.907457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2464.923828] LNet: 118088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.937465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2465.001980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2465.571961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2465.618602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2465.679086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2466.123501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2466.170968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2466.282444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2466.315495] LNet: 118121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.343761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2466.439698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2466.856306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2466.911773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2466.989650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2467.484310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2467.519805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2473.932989] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 13:02:18 (1790701338) [ 2474.078213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2474.112070] LNet: 118729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.159507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2474.206828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2474.231650] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2475.018835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2475.043318] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2476.048080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2476.131431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2476.389425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2476.438786] LNetError: 119009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2476.450109] LNet: Removed LNI 192.168.204.57@tcp [ 2476.468975] LNet: Removed LNI 192.168.204.57@tcp2 [ 2476.508778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2476.593834] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2476.601152] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2476.607072] LNet: 119015:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2477.183767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2477.509612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2477.538521] LNetError: 119052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2477.548047] LNetError: 119052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2477.556729] LNet: Removed LNI 192.168.204.57@tcp [ 2477.573996] LNet: Removed LNI 192.168.204.57@tcp2 [ 2477.616232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2477.721628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2477.745408] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2478.515853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2478.542143] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2479.286175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2479.332478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2479.403167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2479.607086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2479.629928] LNet: 119337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2479.637139] LNet: 119337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2479.644793] LNetError: 119337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2479.652124] LNetError: 119337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2479.661464] LNet: Removed LNI 192.168.204.57@tcp [ 2480.678615] LNet: Removed LNI 192.168.204.57@tcp2 [ 2480.755519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2480.858963] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2480.866656] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2480.871279] LNet: 119343:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2481.235453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2481.364560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2481.382620] LNet: Removed LNI 192.168.204.57@tcp [ 2482.407615] LNet: Removed LNI 192.168.204.57@tcp2 [ 2482.450131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2482.509700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2482.542887] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2483.116315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2483.149090] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2484.099087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2484.180513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2484.429449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2484.486669] LNetError: 119659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.496157] LNetError: 119659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2484.509263] LNet: Removed LNI 192.168.204.57@tcp [ 2484.528722] LNet: Removed LNI 192.168.204.57@tcp2 [ 2484.560968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2484.657410] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2484.669617] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2484.680690] LNet: 119665:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2485.236613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2485.439410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2486.495142] LNet: Removed LNI 192.168.204.57@tcp [ 2486.510081] LNet: Removed LNI 192.168.204.57@tcp2 [ 2486.531586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2486.590400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2486.633516] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2487.439261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2487.456969] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2488.281891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2488.335225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2488.410781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2488.627266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2488.654565] LNet: 119987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.659274] LNet: 119987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2489.695327] LNet: Removed LNI 192.168.204.57@tcp [ 2490.744685] LNet: Removed LNI 192.168.204.57@tcp2 [ 2490.785411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2490.858448] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2490.866312] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2490.877159] LNet: 119993:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2491.408874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2491.610901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2491.637689] LNet: Removed LNI 192.168.204.57@tcp [ 2492.713037] LNet: Removed LNI 192.168.204.57@tcp2 [ 2492.765573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.859237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2492.888622] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2493.618549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2493.677715] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2494.565531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2494.614269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2494.682341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2495.049529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2495.068945] LNetError: 120328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2495.082365] LNetError: 120328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2495.090902] LNet: Removed LNI 192.168.204.57@tcp [ 2495.132261] LNet: Removed LNI 192.168.204.57@tcp2 [ 2495.175228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2495.246622] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2495.258746] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2495.264942] LNet: 120334:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2495.718989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2496.071641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2497.119183] LNet: Removed LNI 192.168.204.57@tcp [ 2497.132252] LNet: Removed LNI 192.168.204.57@tcp2 [ 2497.176707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2497.266836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2497.306090] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2498.055844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2498.082118] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2499.096910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2499.166365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2499.225625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2499.321871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2499.729261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2500.767349] LNet: Removed LNI 192.168.204.57@tcp [ 2501.800059] LNet: Removed LNI 192.168.204.57@tcp2 [ 2501.820853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2501.884655] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2501.891974] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2502.600023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2509.771863] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2511.791753] Lustre: DEBUG MARKER: == sanity-lnet test 411: LNet corrupt fault injection ==== 13:02:55 (1790701375) [ 2511.886708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2511.920303] LNet: 121458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.925470] LNet: 121458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2511.935937] LNetError: 121458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.946589] LNetError: 121458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2511.952096] LNet: Removed LNI 192.168.204.57@tcp [ 2513.005068] LNet: Removed LNI 192.168.204.57@tcp2 [ 2513.088705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2513.198770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2513.247092] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2521.383447] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing lst_setup [ 2523.611432] LNet: 121916:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2523.733502] LNet: 121916:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2523.850437] LNet: 121916:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2523.970878] LNet: 121916:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2524.101163] LNet: 121916:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2524.216287] LNet: 121916:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2524.336331] LNet: 121916:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2524.665788] LNet: 462:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2526.359630] LNet: 38:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2526.525045] LustreError: 71452:0:(brw_test.c:208:brw_check_page()) Bad data in page 0000000015e30a01: 0xeeb0ee5ceeb2eeb3, 0xeeb0eeb1eeb2eeb3 expected [ 2526.551048] LustreError: 71452:0:(brw_test.c:244:brw_check_bulk()) Bulk page 0000000015e30a01 (11/16) is corrupted! [ 2526.567339] LustreError: 71452:0:(brw_test.c:355:brw_client_done_rpc()) Bulk data from 12345-192.168.204.57@tcp is corrupted! [ 2526.946829] LNet: 38:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2527.086879] LustreError: 38:0:(brw_test.c:208:brw_check_page()) Bad data in page 0000000052afd535: 0xeeb0eeb1eeb2eee0, 0xeeb0eeb1eeb2eeb3 expected [ 2527.116264] LustreError: 38:0:(brw_test.c:244:brw_check_bulk()) Bulk page 0000000052afd535 (1/16) is corrupted! [ 2527.138327] LustreError: 38:0:(brw_test.c:408:brw_bulk_ready()) Bulk data from 12345-192.168.204.57@tcp is corrupted! [ 2527.155582] LNet: 38:0:(rpc.c:1294:srpc_send_rpc()) Remote error 74 at 12345-192.168.204.57@tcp, unlink bulk buffer in case peer didn't initiate bulk transfer [ 2527.167559] Lustre: 38:0:(brw_test.c:342:brw_client_done_rpc()) BRW RPC to 12345-192.168.204.57@tcp finished with brw_status: 74 [ 2528.582307] LNet: 462:0:(rpc.c:647:srpc_service_add_buffers()) waiting for adding buffer [ 2542.293052] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing lst_cleanup [ 2542.921129] LNet: 122730:0:(timer.c:222:stt_shutdown()) waiting for 1 threads to terminate [ 2544.299965] LNet: Removed LNI 192.168.204.57@tcp [ 2544.958821] Key type .llcrypt unregistered [ 2544.961292] Key type ._llcrypt unregistered [ 2558.347052] Key type ._llcrypt registered [ 2558.351839] Key type .llcrypt registered [ 2558.890579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2558.923886] alg: No test for adler32 (adler32-zlib) [ 2560.029473] Lustre: Lustre: Build Version: 2.17.59_2_ga235f42 [ 2560.305868] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2562.071260] Key type lgssc registered [ 2563.566482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2673.353257] Lustre: Mounted lustre-client - version 2.17.59_2_ga235f42 [ 2679.144055] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2691.123694] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 13:05:53 (1790701553) [ 2691.400885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2694.581838] Lustre: DEBUG MARKER: sanity-lnet test_450: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2699.242664] Lustre: lustre-OST0000-osc-ffff8f8cc7fce800: disconnect after 23s idle [ 2704.049712] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 13:06:07 (1790701567) [ 2704.210955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2705.764898] Lustre: DEBUG MARKER: sanity-lnet test_475: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2712.499291] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 13:06:16 (1790701576) [ 2712.636602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2714.135934] Lustre: DEBUG MARKER: sanity-lnet test_476: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2719.290771] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:06:23 (1790701583) [ 2719.344132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2720.930273] Lustre: DEBUG MARKER: sanity-lnet test_500: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2726.315090] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:06:30 (1790701590) [ 2726.619333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2728.042199] Lustre: DEBUG MARKER: sanity-lnet test_501: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2733.877198] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:06:38 (1790701598) [ 2734.164688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2735.774676] Lustre: DEBUG MARKER: sanity-lnet test_502: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2741.523426] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 13:06:45 (1790701605) [ 2741.661969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2743.593670] Lustre: DEBUG MARKER: sanity-lnet test_525: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2749.917046] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 13:06:53 (1790701613) [ 2752.095579] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2753.996502] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 13:06:58 (1790701618) [ 2763.080198] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 13:07:07 (1790701627) [ 2772.525893] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 13:07:16 (1790701636) [ 2781.731257] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 13:07:25 (1790701645) [ 2790.392285] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 13:07:34 (1790701654) [ 2799.037256] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 13:07:43 (1790701663) [ 2809.132249] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 13:07:53 (1790701673) [ 2819.667532] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 13:08:03 (1790701683) [ 2829.606238] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 13:08:13 (1790701693) [ 2837.449550] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 13:08:21 (1790701701) [ 2845.486097] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 13:08:29 (1790701709) [ 2853.363274] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 13:08:37 (1790701717) [ 2861.734182] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 13:08:45 (1790701725) [ 2870.765260] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 13:08:54 (1790701734) [ 2879.964023] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 13:09:03 (1790701743) [ 2887.471770] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 13:09:11 (1790701751) [ 2897.874480] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 13:09:21 (1790701761) [ 2906.541271] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 13:09:30 (1790701770) [ 2906.838648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2908.589507] Lustre: DEBUG MARKER: sanity-lnet test_617: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2914.837984] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 13:09:39 (1790701779) [ 2924.673361] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 13:09:48 (1790701788) [ 2934.130759] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 13:09:57 (1790701797) [ 2943.510941] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 13:10:07 (1790701807) [ 2952.437987] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 13:10:15 (1790701815) [ 2961.137657] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 13:10:25 (1790701825) [ 2970.467542] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 13:10:34 (1790701834) [ 2979.562401] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 13:10:43 (1790701843) [ 2987.614432] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 13:10:51 (1790701851) [ 2996.478150] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 13:11:00 (1790701860) [ 3004.935242] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 13:11:08 (1790701868) [ 3014.826942] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 13:11:18 (1790701878) [ 3023.470961] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2706 sec ========== 13:11:27 (1790701887) [ 3026.023717] Lustre: Unmounted lustre-client [ 3088.755643] Key type lgssc unregistered [ 3089.190178] LNet: 148331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3089.205141] LNetError: 148331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3089.233366] LNet: Removed LNI 192.168.204.57@tcp [ 3090.195226] Key type .llcrypt unregistered [ 3090.197605] Key type ._llcrypt unregistered