[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 443691763 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002382] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007648] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010126] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013095] *** VALIDATE selinux *** [ 0.021698] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026725] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028087] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030012] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032067] *** VALIDATE tmpfs *** [ 0.033507] *** VALIDATE proc *** [ 0.035021] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039054] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041032] Spectre V2 : User space: Vulnerable [ 0.043004] Speculative Store Bypass: Vulnerable [ 0.046350] debug: unmapping init [mem 0xffffffffaf459000-0xffffffffaf460fff] [ 0.048205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049752] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050023] ... version: 2 [ 0.051010] ... bit width: 48 [ 0.052011] ... generic registers: 4 [ 0.053014] ... value mask: 0000ffffffffffff [ 0.054016] ... max period: 00007fffffffffff [ 0.055012] ... fixed-purpose events: 3 [ 0.056009] ... event mask: 000000070000000f [ 0.057333] rcu: Hierarchical SRCU implementation. [ 0.059510] smp: Bringing up secondary CPUs ... [ 0.060600] x86: Booting SMP configuration: [ 0.061034] .... node #0, CPUs: #1 #2 #3 [ 0.064640] smp: Brought up 1 node, 4 CPUs [ 0.066014] smpboot: Max logical packages: 1 [ 0.067019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141000] node 0 deferred pages initialised in 71ms [ 0.145008] devtmpfs: initialized [ 0.146000] x86/mm: Memory block size: 128MB [ 0.149067] gcov: version magic: 0x41383552 [ 0.152310] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157317] pinctrl core: initialized pinctrl subsystem [ 0.159245] [ 0.159716] ************************************************************* [ 0.160011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162015] ** ** [ 0.165011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167010] ** ** [ 0.169010] ** This means that this kernel is built to expose internal ** [ 0.171012] ** IOMMU data structures, which may compromise security on ** [ 0.173013] ** your system. ** [ 0.175013] ** ** [ 0.177011] ** If you see this message and you are not debugging the ** [ 0.179108] ** kernel, report this immediately to your vendor! ** [ 0.182011] ** ** [ 0.184010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186013] ************************************************************* [ 0.188503] NET: Registered protocol family 16 [ 0.190522] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199111] cpuidle: using governor menu [ 0.200647] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201439] PCI: Using configuration type 1 for base access [ 0.202098] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.219035] cryptd: max_cpu_qlen set to 1000 [ 0.222287] ACPI: Added _OSI(Module Device) [ 0.224014] ACPI: Added _OSI(Processor Device) [ 0.226014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.228016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232212] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.239120] ACPI: Interpreter enabled [ 0.241062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.242013] ACPI: Using IOAPIC for interrupt routing [ 0.244148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247396] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257482] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266124] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270663] acpiphp: Slot [2] registered [ 0.272122] acpiphp: Slot [5] registered [ 0.274188] acpiphp: Slot [6] registered [ 0.275068] acpiphp: Slot [3] registered [ 0.276093] acpiphp: Slot [4] registered [ 0.277084] acpiphp: Slot [7] registered [ 0.279117] acpiphp: Slot [8] registered [ 0.280088] acpiphp: Slot [9] registered [ 0.282095] acpiphp: Slot [10] registered [ 0.283090] acpiphp: Slot [11] registered [ 0.284109] acpiphp: Slot [12] registered [ 0.286088] acpiphp: Slot [13] registered [ 0.287099] acpiphp: Slot [14] registered [ 0.289118] acpiphp: Slot [15] registered [ 0.290111] acpiphp: Slot [16] registered [ 0.292108] acpiphp: Slot [17] registered [ 0.293094] acpiphp: Slot [18] registered [ 0.295103] acpiphp: Slot [19] registered [ 0.296104] acpiphp: Slot [20] registered [ 0.298131] acpiphp: Slot [21] registered [ 0.299122] acpiphp: Slot [22] registered [ 0.301161] acpiphp: Slot [23] registered [ 0.302092] acpiphp: Slot [24] registered [ 0.304095] acpiphp: Slot [25] registered [ 0.305091] acpiphp: Slot [26] registered [ 0.306100] acpiphp: Slot [27] registered [ 0.308088] acpiphp: Slot [28] registered [ 0.309083] acpiphp: Slot [29] registered [ 0.310080] acpiphp: Slot [30] registered [ 0.311087] acpiphp: Slot [31] registered [ 0.313055] PCI host bridge to bus 0000:00 [ 0.314018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.316019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.319034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.322022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.324023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326130] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.327934] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.332336] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.338616] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.343142] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.345017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.347017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.349019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351635] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.355092] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.357040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.359893] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.364014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.376016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384915] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.396021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.414023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.430014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439512] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.453017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.472022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.482908] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.485399] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.487248] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.488189] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.490122] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.494102] iommu: Default domain type: Passthrough [ 0.495432] SCSI subsystem initialized [ 0.496133] ACPI: bus type USB registered [ 0.497312] usbcore: registered new interface driver usbfs [ 0.500067] usbcore: registered new interface driver hub [ 0.501078] usbcore: registered new device driver usb [ 0.503136] pps_core: LinuxPPS API ver. 1 registered [ 0.505010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.508081] PTP clock support registered [ 0.510124] EDAC MC: Ver: 3.0.0 [ 0.512124] PCI: Using ACPI for IRQ routing [ 0.513590] NetLabel: Initializing [ 0.514008] NetLabel: domain hash size = 128 [ 0.515008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.517068] NetLabel: unlabeled traffic allowed by default [ 0.518253] vgaarb: loaded [ 0.519164] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.521011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.528000] clocksource: Switched to clocksource kvm-clock [ 0.637704] VFS: Disk quotas dquot_6.6.0 [ 0.639139] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.641563] *** VALIDATE ramfs *** [ 0.642728] *** VALIDATE hugetlbfs *** [ 0.644976] pnp: PnP ACPI init [ 0.647483] pnp: PnP ACPI: found 6 devices [ 0.665675] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.669072] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.671157] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.673627] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.676235] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.678751] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.682068] NET: Registered protocol family 2 [ 0.684740] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.689388] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.693134] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.698755] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.702362] TCP: Hash tables configured (established 65536 bind 65536) [ 0.705472] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.708639] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.710759] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.713469] NET: Registered protocol family 1 [ 0.715365] RPC: Registered named UNIX socket transport module. [ 0.716964] RPC: Registered udp transport module. [ 0.718031] RPC: Registered tcp transport module. [ 0.719054] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.720539] NET: Registered protocol family 44 [ 0.722049] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.723839] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.726075] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.728578] PCI: CLS 0 bytes, default 64 [ 0.730406] Unpacking initramfs... [ 2.035274] debug: unmapping init [mem 0xffff893ebcc64000-0xffff893ebffcffff] [ 2.038555] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.039900] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.041695] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.468376] Initialise system trusted keyrings [ 2.469420] Key type blacklist registered [ 2.470594] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.476377] zbud: loaded [ 2.478367] *** VALIDATE nfs *** [ 2.479145] *** VALIDATE nfs4 *** [ 2.480279] pstore: using deflate compression [ 2.483196] Platform Keyring initialized [ 2.577145] NET: Registered protocol family 38 [ 2.578269] Key type asymmetric registered [ 2.579151] Asymmetric key parser 'x509' registered [ 2.580641] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.582428] io scheduler mq-deadline registered [ 2.583530] io scheduler kyber registered [ 2.584649] io scheduler bfq registered [ 2.586350] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.588534] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.590520] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.592383] ACPI: Power Button [PWRF] [ 2.596911] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.602904] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.612435] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.639607] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.667335] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.671436] Non-volatile memory driver v1.3 [ 2.672838] Linux agpgart interface v0.103 [ 2.699969] virtio_blk virtio1: [vda] 150608 512-byte logical blocks (77.1 MB/73.5 MiB) [ 2.702323] vda: detected capacity change from 0 to 77111296 [ 2.720039] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.722184] vdb: detected capacity change from 0 to 1073741824 [ 2.731736] libphy: Fixed MDIO Bus: probed [ 2.741262] usbcore: registered new interface driver usbserial_generic [ 2.743244] usbserial: USB Serial support registered for generic [ 2.745105] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.748754] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.750193] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.752118] mousedev: PS/2 mouse device common for all mice [ 2.754690] rtc_cmos 00:05: RTC can wake from S4 [ 2.758166] rtc_cmos 00:05: registered as rtc0 [ 2.759911] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.762756] intel_pstate: CPU model not supported [ 2.765130] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.769125] hid: raw HID events driver (C) Jiri Kosina [ 2.770031] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.771526] usbcore: registered new interface driver usbhid [ 2.775285] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.776959] usbhid: USB HID core driver [ 2.777122] drop_monitor: Initializing network drop monitor service [ 2.783552] Initializing XFRM netlink socket [ 2.785182] NET: Registered protocol family 10 [ 2.788204] Segment Routing with IPv6 [ 2.789822] NET: Registered protocol family 17 [ 2.791583] mpls_gso: MPLS GSO support [ 2.796499] RAS: Correctable Errors collector initialized. [ 2.798149] AVX version of gcm_enc/dec engaged. [ 2.799499] AES CTR mode by8 optimization enabled [ 2.872000] sched_clock: Marking stable (2871976454, 0)->(3775005070, -903028616) [ 2.874652] registered taskstats version 1 [ 2.876229] Loading compiled-in X.509 certificates [ 2.877896] zswap: loaded using pool lzo/zbud [ 2.900633] Key type big_key registered [ 2.911790] Key type encrypted registered [ 2.913111] ima: No TPM chip found, activating TPM-bypass! [ 2.914783] ima: Allocated hash algorithm: sha1 [ 2.916061] ima: No architecture policies found [ 2.917399] evm: Initialising EVM extended attributes: [ 2.918863] evm: security.selinux [ 2.919759] evm: security.ima [ 2.920514] evm: security.capability [ 2.921445] evm: HMAC attrs: 0x1 [ 2.923340] rtc_cmos 00:05: setting system clock to 2026-10-01 23:33:37 UTC (1790897617) [ 2.929723] debug: unmapping init [mem 0xffffffffb0403000-0xffffffffb05fffff] [ 2.932275] debug: unmapping init [mem 0xffffffffaf182000-0xffffffffaf458fff] [ 2.937066] Write protecting the kernel read-only data: 28672k [ 2.939941] debug: unmapping init [mem 0xffffffffad803000-0xffffffffad9fffff] [ 2.942197] debug: unmapping init [mem 0xffffffffae114000-0xffffffffae1fffff] [ 2.969643] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.976018] systemd[1]: Detected virtualization kvm. [ 2.977605] systemd[1]: Detected architecture x86-64. [ 2.979062] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.006025] systemd[1]: No hostname configured. [ 3.007343] systemd[1]: Set hostname to . [ 3.009165] random: systemd: uninitialized urandom read (16 bytes read) [ 3.011068] systemd[1]: Initializing machine ID from random generator. [ 3.047518] random: ln: uninitialized urandom read (6 bytes read) [ 3.136840] random: systemd: uninitialized urandom read (16 bytes read) [ 3.138562] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.142073] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.147088] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.664875] device-mapper: uevent: version 1.0.3 [ 3.666864] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 4.266151] random: fast init done [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.287473] virtio_net virtio0 ens2: renamed from eth0 [ 4.313237] scsi host0: ata_piix [ 4.357721] scsi host1: ata_piix [ 4.359263] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.361986] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.880047] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.427016] random: crng init done [ 9.428442] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.651622] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.641255] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.863124] SELinux: Disabled at runtime. [ 10.922074] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.927343] systemd[1]: Detected virtualization kvm. [ 10.928580] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.290120] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.293236] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.297663] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.301510] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.305067] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.312077] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.320259] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ 11.392108] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 11.800064] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.041242] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.050141] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.152698] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.161105] EDAC sbridge: Ver: 1.1.2 [ 13.291248] Key type dns_resolver registered [ 13.582120] NFS: Registering the id_resolver key type [ 13.583976] Key type id_resolver registered [ 13.585054] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg159-client login: [ 39.886985] libcfs: loading out-of-tree module taints kernel. [ 39.945106] Key type ._llcrypt registered [ 39.950326] Key type .llcrypt registered [ 40.367042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 40.382132] alg: No test for adler32 (adler32-zlib) [ 41.391307] Lustre: Lustre: Build Version: 2.17.59_2_g8e10c0d [ 41.714912] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 43.367325] Key type lgssc registered [ 44.110528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 180.582976] hrtimer: interrupt took 8734029 ns [ 218.733586] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d [ 224.681165] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.654893] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing check_logdir /tmp/testlogs/ [ 244.203876] Lustre: lustre-OST0000-osc-ffff893f05465800: disconnect after 23s idle [ 247.690682] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing yml_node [ 252.310176] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 254.829831] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 257.385461] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 259.259538] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Oct 1 19:37:52 EDT 2026 [ 276.414323] Lustre: Unmounted lustre-client [ 332.895455] Key type lgssc unregistered [ 333.322051] LNet: 9198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.334181] LNetError: 9198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 333.351935] LNet: Removed LNI 192.168.201.59@tcp [ 334.111322] Key type .llcrypt unregistered [ 334.113490] Key type ._llcrypt unregistered [ 369.808369] Key type ._llcrypt registered [ 369.811530] Key type .llcrypt registered [ 370.056746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 370.092785] alg: No test for adler32 (adler32-zlib) [ 371.177704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 371.256043] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 371.312931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 376.312197] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 376.464978] LNet: 10135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.474249] LNetError: 10135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 376.512584] LNet: Removed LNI 192.168.201.59@tcp [ 377.320877] Key type .llcrypt unregistered [ 377.324523] Key type ._llcrypt unregistered [ 384.892209] Key type ._llcrypt registered [ 384.895216] Key type .llcrypt registered [ 386.086094] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:39:59 (1790897999) [ 386.589199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 386.596832] alg: No test for adler32 (adler32-zlib) [ 387.567929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.636434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.666382] LNet: 10557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.690826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.775479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 387.798830] LNet: 10564:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 394.558899] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:40:07 (1790898007) [ 394.812306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.839229] LNet: 11149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.858959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.941396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 402.515081] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:40:15 (1790898015) [ 403.069385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.121184] LNet: 11742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.143990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.250851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 410.690681] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:40:23 (1790898023) [ 411.191912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.241966] LNet: 12336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.261686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.398160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 419.199711] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:40:31 (1790898031) [ 419.304331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.367500] LNet: 12924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.410946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.538935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 419.941996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 426.851299] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:40:39 (1790898039) [ 427.194920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.252495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.310757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 427.838779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 428.388621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 435.198145] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:40:48 (1790898048) [ 435.303713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.331810] LNet: 14132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.339741] LNet: 14132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 435.389346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.427250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 442.154545] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:40:55 (1790898055) [ 442.282660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.333144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.372951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 442.405404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 442.757126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 442.805656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 443.274525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 443.325138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 443.706673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 443.754791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 444.122406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 444.172408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 444.596515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 444.644825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 445.225697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 445.285652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 445.789807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 445.834406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 446.336463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 446.372102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 446.800435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 446.849329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 447.261387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 447.289890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 447.753736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 447.806178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 448.263294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 448.304989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 454.721855] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:41:08 (1790898068) [ 454.978460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.997346] LNet: 15416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.002650] LNet: 15416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 455.025243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.082803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 455.114484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 462.052746] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:41:14 (1790898074) [ 462.186215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.261499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.399116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 462.444837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 471.376549] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:41:24 (1790898084) [ 471.497786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.606282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.759767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 471.802332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 478.773764] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:41:31 (1790898091) [ 478.921457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.032967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.112977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 479.147717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 486.016620] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:41:39 (1790898099) [ 486.253135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.289911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.353918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 486.412083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 493.226939] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:41:46 (1790898106) [ 493.337974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.384993] LNet: 18381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.392158] LNet: 18381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 493.443599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.532837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 493.577525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 500.478807] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:41:53 (1790898113) [ 500.676821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.727803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.824977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 501.667214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.538936] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:42:03 (1790898123) [ 510.694625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.725924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.971631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.278434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.805282] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:42:12 (1790898132) [ 519.951319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.041365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.241810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 520.676731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 528.919596] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:42:21 (1790898141) [ 529.046052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.109448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.821527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 530.143672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 539.608951] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:42:32 (1790898152) [ 539.954041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.984164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.086734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.173231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 547.950505] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:42:40 (1790898160) [ 548.200447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.255201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.309616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 548.356497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 556.046994] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:42:49 (1790898169) [ 556.536126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.611100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.726967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 564.615598] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:42:57 (1790898177) [ 565.063880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.119295] LNet: 23306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.127291] LNet: 23306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 565.181210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.277180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 574.216626] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:43:07 (1790898187) [ 574.368850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.463933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.568941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 575.030194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 583.410468] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:43:16 (1790898196) [ 583.669879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.705607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.785290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 583.840530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 591.785210] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:43:24 (1790898204) [ 592.184945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.248322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.358738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 592.437449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 599.867678] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:43:33 (1790898213) [ 600.233336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.295522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.420988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 600.470507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 609.215408] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:43:41 (1790898221) [ 609.328364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.396971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.498566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 609.534292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 617.253718] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:43:50 (1790898230) [ 617.349366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.409443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.510902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 617.618685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 617.640301] LNetError: 26887:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 617.709560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 617.756562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 624.905953] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:43:57 (1790898237) [ 625.009949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.068598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.187444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VoGWDsqW1D [ 625.254423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VoGWDsqW1D [ 625.306067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VoGWDsqW1D [ 633.273547] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:44:06 (1790898246) [ 633.760363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.841578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.914537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 633.966101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 634.065684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 645.859450] Lustre: DEBUG MARKER: == sanity-lnet test 29: peer show: unknown peer should be reported ========================================================== 19:44:17 (1790898257) [ 646.018714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.144256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.705458] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 19:44:28 (1790898268) [ 656.862742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.967217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.221844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 657.280648] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 659.433759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 661.101987] Lustre: Lustre: Build Version: 2.17.59_2_g8e10c0d [ 662.999518] Key type lgssc registered [ 664.692577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 668.379174] LNetError: 29257:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.59@tcp99 has no available nets [ 671.584111] Key type lgssc unregistered [ 671.904553] LNetError: 30403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.921598] LNet: Removed LNI 192.168.201.59@tcp [ 672.479823] Key type .llcrypt unregistered [ 672.482128] Key type ._llcrypt unregistered [ 679.632572] Key type ._llcrypt registered [ 679.634976] Key type .llcrypt registered [ 680.939356] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:44:54 (1790898294) [ 681.420492] Key type .llcrypt unregistered [ 681.422970] Key type ._llcrypt unregistered [ 689.219863] Key type ._llcrypt registered [ 689.222395] Key type .llcrypt registered [ 689.457955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 689.479707] alg: No test for adler32 (adler32-zlib) [ 690.441391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.503341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 690.522665] LNet: 31335:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 690.530835] LNet: 31335:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 690.536097] LNet: 31335:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 690.547297] LNet: Message forwarding will be enabled in 2s [ 693.663169] LNet: Message forwarding enabled [ 694.179646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 694.234886] LNet: Message forwarding will stop in 2s to 6s [ 697.762629] LNet: Message forwarding disabled [ 698.738549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 698.772804] LNet: 31433:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 698.785886] LNet: 31433:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 698.794850] LNet: 31433:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 698.812696] LNet: Message forwarding will be enabled in 2s [ 698.994274] LNet: 31446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.659808] Key type .llcrypt unregistered [ 699.663067] Key type ._llcrypt unregistered [ 712.451517] Key type ._llcrypt registered [ 712.459706] Key type .llcrypt registered [ 713.742266] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:45:26 (1790898326) [ 714.118427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 714.140113] alg: No test for adler32 (adler32-zlib) [ 715.166696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.246613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 721.961604] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:45:35 (1790898335) [ 722.118246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.146351] LNet: 32866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.200240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.274694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 722.320268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 722.352939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 722.382220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 722.417525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 722.444961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 722.466442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 722.507723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 722.562213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 722.610635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 722.655628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 722.695799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 722.718694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 722.748498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 722.780553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 722.814125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 722.854384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 722.925327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 722.964910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 722.996365] LNetError: 32911:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 729.448361] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:45:42 (1790898342) [ 729.673728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.705542] LNet: 33490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.731422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.801497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 737.281933] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:45:50 (1790898350) [ 739.670859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg159-server [ 742.245712] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:45:54 (1790898354) [ 742.339249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.387755] LNet: 34430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.447455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.570243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.614314] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 743.617343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[60-64]@tcp [ 743.649702] LNet: 34569:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 743.706152] LNetError: 34440:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.60@tcp is being used as a gateway but routing feature is not turned on [ 743.720237] LNetError: 34440:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.60@tcp has gone from up to down [ 746.721035] LNetError: 34444:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.62@tcp has gone from up to down [ 751.772569] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:46:04 (1790898364) [ 753.664643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg159-server [ 755.276564] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:46:08 (1790898368) [ 755.491933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.513798] LNet: 35520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.517892] LNetError: 35520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 756.583056] LNet: Removed LNI 192.168.201.59@tcp [ 756.614296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.681972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.717442] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 757.381479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[60-65/2]@tcp [ 757.428596] LNet: 35637:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 757.439283] LNet: 35637:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 757.474673] LNetError: 35524:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.60@tcp is being used as a gateway but routing feature is not turned on [ 757.480362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[60-65/2]@tcp [ 757.493156] LNetError: 35524:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 757.510755] LNetError: 35524:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.60@tcp has gone from up to down [ 757.526684] LNetError: 35524:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 764.963875] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:46:18 (1790898378) [ 765.089509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.108177] LNet: 36233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.114228] LNetError: 36233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.149050] LNet: Removed LNI 192.168.201.59@tcp [ 765.214176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.300666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 765.361391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.390978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.456325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 765.509535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.551872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.607911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 765.679718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.713888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.773111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 765.850132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.912484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.980909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 766.080358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.109424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.158361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 766.216660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.255657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.292573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 766.344979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.373772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.432156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 766.538634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.586353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.633651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 766.716932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.752994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.807603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 766.898949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.930896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.975861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 767.021300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 767.046664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.140404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 774.476318] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:46:27 (1790898387) [ 776.390646] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg159-server [ 778.300383] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:46:31 (1790898391) [ 780.439151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg159-server [ 782.766923] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:46:35 (1790898395) [ 782.835909] LNet: 37649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.839407] LNet: 37649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 783.492463] Key type .llcrypt unregistered [ 783.495581] Key type ._llcrypt unregistered [ 789.364722] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 789.663304] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 789.687867] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 789.695945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 791.372375] Key type ._llcrypt registered [ 791.377360] Key type .llcrypt registered [ 791.633661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 791.653571] alg: No test for adler32 (adler32-zlib) [ 792.648646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.733719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 792.762176] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 793.768326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 793.800217] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 796.871726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 796.896613] LNet: Removed LNI 10.1.2.3@tcp [ 797.069387] LNet: 38370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.076196] LNetError: 38370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.094227] LNet: Removed LNI 192.168.201.59@tcp [ 797.911678] Key type .llcrypt unregistered [ 797.913676] Key type ._llcrypt unregistered [ 809.715686] Key type ._llcrypt registered [ 809.718896] Key type .llcrypt registered [ 811.204576] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:47:04 (1790898424) [ 811.791458] Key type .llcrypt unregistered [ 811.798660] Key type ._llcrypt unregistered [ 818.243931] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 818.537682] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 818.555361] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 818.562426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 821.235104] Key type ._llcrypt registered [ 821.237486] Key type .llcrypt registered [ 821.538876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.564469] alg: No test for adler32 (adler32-zlib) [ 822.470902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.515172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.535384] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 823.142228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 823.166461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.757951] LNet: 39876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.765613] LNetError: 39876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 826.772089] LNet: Removed LNI 10.1.2.3@tcp [ 826.789382] LNet: Removed LNI 192.168.201.59@tcp [ 827.602583] Key type .llcrypt unregistered [ 827.606820] Key type ._llcrypt unregistered [ 843.196624] Key type ._llcrypt registered [ 843.203744] Key type .llcrypt registered [ 844.903954] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:47:37 (1790898457) [ 845.434296] Key type .llcrypt unregistered [ 845.437675] Key type ._llcrypt unregistered [ 852.328906] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 852.651565] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 852.659415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 852.673590] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 855.808399] Key type ._llcrypt registered [ 855.812746] Key type .llcrypt registered [ 856.113254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.145783] alg: No test for adler32 (adler32-zlib) [ 857.180928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.271896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 857.324334] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 860.007042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 860.068683] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 862.565414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 862.619408] LNet: Removed LNI 10.1.2.3@tcp [ 862.635621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 862.670561] LNet: Removed LNI 10.1.2.31@tcp [ 862.700528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 862.740646] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 863.940831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 863.961145] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 865.329803] LNet: 41816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.341108] LNetError: 41816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 865.350478] LNet: Removed LNI 10.1.2.31@tcp [ 865.366915] LNet: Removed LNI 10.1.2.3@tcp [ 866.036734] Key type .llcrypt unregistered [ 866.043584] Key type ._llcrypt unregistered [ 881.134496] Key type ._llcrypt registered [ 881.136066] Key type .llcrypt registered [ 882.654826] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:48:15 (1790898495) [ 883.017945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.028931] alg: No test for adler32 (adler32-zlib) [ 883.975565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.053707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.083958] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 885.072251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.118890] LNet: 42792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.125311] LNetError: 42792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.156189] LNet: Removed LNI 192.168.201.59@tcp [ 885.218076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.327528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.59@tcp [ 885.384533] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 886.609784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.59@tcp [ 886.670663] LNet: Removed LNI 192.168.201.59@tcp [ 886.759436] LNet: 42928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.764703] Key type .llcrypt unregistered [ 887.768785] Key type ._llcrypt unregistered [ 900.456272] Key type ._llcrypt registered [ 900.464646] Key type .llcrypt registered [ 902.320807] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:48:34 (1790898514) [ 904.558450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg159-server [ 906.341667] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:48:39 (1790898519) [ 907.066290] Key type .llcrypt unregistered [ 907.068087] Key type ._llcrypt unregistered [ 914.100787] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 914.269545] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 914.280471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 914.293253] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 916.959897] Key type ._llcrypt registered [ 916.963967] Key type .llcrypt registered [ 917.157152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.167630] alg: No test for adler32 (adler32-zlib) [ 918.117485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.190089] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 918.983739] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 920.958473] LNet: 44785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.968206] LNetError: 44785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.973326] LNet: Removed LNI 192.168.201.59@tcp1 [ 920.985526] LNet: Removed LNI 10.1.2.3@tcp2 [ 921.702933] Key type .llcrypt unregistered [ 921.704661] Key type ._llcrypt unregistered [ 934.857745] Key type ._llcrypt registered [ 934.860360] Key type .llcrypt registered [ 936.411964] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:49:09 (1790898549) [ 936.871187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.900679] alg: No test for adler32 (adler32-zlib) [ 937.940576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.081979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 938.134137] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 939.214467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.240846] LNet: 45762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.250198] LNetError: 45762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.277493] LNet: Removed LNI 192.168.201.59@tcp [ 939.311424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.387388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 939.427640] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 947.788503] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:49:20 (1790898560) [ 947.990452] LNet: 46354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.017971] LNetError: 46354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.051567] LNet: Removed LNI 192.168.201.59@tcp [ 949.044664] Key type .llcrypt unregistered [ 949.049912] Key type ._llcrypt unregistered [ 955.819668] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 956.194488] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 956.204279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 956.213187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 958.920677] Key type ._llcrypt registered [ 958.923906] Key type .llcrypt registered [ 959.134834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.159777] alg: No test for adler32 (adler32-zlib) [ 960.187032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.288572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.313442] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 961.202568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 961.240457] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 963.190835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.217167] LNet: 47105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.223938] LNetError: 47105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 963.231721] LNet: Removed LNI 10.1.2.3@tcp [ 963.251569] LNet: Removed LNI 192.168.201.59@tcp [ 963.271246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.350217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 963.400410] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 963.406152] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 963.881967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.895379] LNet: 47132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.900878] LNetError: 47132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 963.906562] LNetError: 47132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 963.910557] LNet: Removed LNI 10.1.2.3@tcp [ 963.925201] LNet: Removed LNI 192.168.201.59@tcp [ 963.968832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.033888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 964.049393] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 964.054170] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 964.484728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 964.521817] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 965.426650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.460193] LNet: 47294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.465655] LNetError: 47294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.474243] LNetError: 47294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 965.481794] LNet: Removed LNI 10.1.2.3@tcp [ 965.483820] LNet: Removed LNI 192.168.201.59@tcp [ 965.501767] LNet: Removed LNI 192.168.201.59@tcp2 [ 965.530670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.624523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 965.657941] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 965.671237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.679464] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 972.690763] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:49:46 (1790898586) [ 972.874937] LNet: 47899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.880594] LNetError: 47899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 972.885856] LNetError: 47899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 972.895471] LNet: Removed LNI 10.1.2.3@tcp [ 972.897129] LNet: Removed LNI 192.168.201.59@tcp [ 972.906328] LNet: Removed LNI 192.168.201.59@tcp2 [ 973.428582] Key type .llcrypt unregistered [ 973.430803] Key type ._llcrypt unregistered [ 980.010739] Key type ._llcrypt registered [ 980.014199] Key type .llcrypt registered [ 980.317728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 980.347727] alg: No test for adler32 (adler32-zlib) [ 981.326780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.437384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 981.483196] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 983.198301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.261623] LNet: 48294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.283333] LNetError: 48294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.303789] LNet: Removed LNI 192.168.201.59@tcp [ 983.340382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.435970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 983.461313] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 984.198221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.260220] LNet: 48324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.270828] LNetError: 48324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.299718] LNet: Removed LNI 192.168.201.59@tcp [ 984.330411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.446074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 984.497951] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 984.957732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.009995] LNet: Removed LNI 192.168.201.59@tcp [ 985.032337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.131394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 985.176367] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 986.087310] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 986.416541] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 986.426601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 986.434548] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 988.992622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.013559] LNet: 48433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.020804] LNet: 48433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 989.029681] LNetError: 48433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.041142] LNetError: 48433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 989.062185] LNet: Removed LNI 192.168.201.59@tcp [ 989.097674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.211691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 989.261466] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1000.463324] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:50:13 (1790898613) [ 1000.703320] LNet: 49037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.714221] LNetError: 49037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.781777] LNet: Removed LNI 192.168.201.59@tcp [ 1001.576522] Key type .llcrypt unregistered [ 1001.578815] Key type ._llcrypt unregistered [ 1011.318370] Key type ._llcrypt registered [ 1011.320198] Key type .llcrypt registered [ 1011.491501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.502976] alg: No test for adler32 (adler32-zlib) [ 1012.541299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.589513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1012.615056] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1013.835738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.847460] LNet: 49439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.849931] LNetError: 49439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.862523] LNet: Removed LNI 192.168.201.59@tcp [ 1013.878674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.930444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1013.960479] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1022.764346] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:50:35 (1790898635) [ 1023.107107] LNet: 50039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.114992] LNetError: 50039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.139705] LNet: Removed LNI 192.168.201.59@tcp [ 1024.232555] Key type .llcrypt unregistered [ 1024.235784] Key type ._llcrypt unregistered [ 1031.189542] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1031.548475] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1031.555322] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1031.557780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1033.202842] Key type ._llcrypt registered [ 1033.206882] Key type .llcrypt registered [ 1033.547934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1033.582607] alg: No test for adler32 (adler32-zlib) [ 1034.599847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.719904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1034.764366] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1036.226953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.266780] LNet: 50483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.278454] LNetError: 50483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.311779] LNet: Removed LNI 192.168.201.59@tcp [ 1036.353719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.459536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1036.498884] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1044.590987] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:50:57 (1790898657) [ 1044.852744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1044.885412] LNet: 51091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.890742] LNetError: 51091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.908644] LNet: Removed LNI 192.168.201.59@tcp [ 1044.927290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.063955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1053.110643] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:51:05 (1790898665) [ 1053.268828] LNet: 51680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.050757] Key type .llcrypt unregistered [ 1054.060140] Key type ._llcrypt unregistered [ 1061.580071] Key type ._llcrypt registered [ 1061.585401] Key type .llcrypt registered [ 1061.907078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.931527] alg: No test for adler32 (adler32-zlib) [ 1062.996981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.094588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1063.136096] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1064.165176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1064.186372] LNet: 52080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.193241] LNetError: 52080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.220818] LNet: Removed LNI 192.168.201.59@tcp [ 1064.245844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.348240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1064.405351] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1072.077063] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:51:25 (1790898685) [ 1072.228802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1072.271416] LNet: 52677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.283295] LNetError: 52677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.323453] LNet: Removed LNI 192.168.201.59@tcp [ 1072.399246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.497568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1082.172598] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:51:34 (1790898694) [ 1084.074655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1086.337750] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:51:39 (1790898699) [ 1088.691103] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1090.515512] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:51:43 (1790898703) [ 1092.744144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1094.764880] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:51:47 (1790898707) [ 1096.799171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1099.041161] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:51:51 (1790898711) [ 1101.038761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1102.875683] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 19:51:56 (1790898716) [ 1103.499348] LNet: 54733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.516619] Key type .llcrypt unregistered [ 1104.518253] Key type ._llcrypt unregistered [ 1112.443533] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1112.695979] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1112.703489] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1112.711497] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1115.297972] Key type ._llcrypt registered [ 1115.302437] Key type .llcrypt registered [ 1115.567351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1115.581446] alg: No test for adler32 (adler32-zlib) [ 1116.593685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1116.776605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1116.800580] LNet: Added LNI 192.168.201.59@tcp [8/512/0/180] [ 1116.819399] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1116.825096] LNet: 55055:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1116.949686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1116.993184] LNet: 55074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.008675] LNetError: 55074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1117.016611] LNet: Removed LNI 10.1.2.3@tcp [ 1117.018213] LNetError: 55074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1117.041376] LNet: Removed LNI 192.168.201.59@tcp [ 1117.055390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1117.162932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1117.201594] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1117.296692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1117.314980] LNet: 55098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.319554] LNetError: 55098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1117.341907] LNet: Removed LNI 192.168.201.59@tcp [ 1117.354706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1117.399428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1117.422887] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1117.435515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1117.437905] LNet: 55106:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1117.529329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1117.561858] LNet: 55124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.584552] LNetError: 55124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1117.595330] LNet: Removed LNI 10.1.2.3@tcp [ 1117.597773] LNetError: 55124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1117.615853] LNet: Removed LNI 192.168.201.59@tcp [ 1117.650360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1117.766487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1117.803718] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1117.816958] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1117.819024] LNet: 55132:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1118.112654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1118.129617] LNet: 55158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.139822] LNetError: 55158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1118.149334] LNet: Removed LNI 10.1.2.3@tcp [ 1118.153059] LNetError: 55158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.174930] LNet: Removed LNI 192.168.201.59@tcp [ 1118.202776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1118.331663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1118.393363] LNet: Added LNI 192.168.201.59@tcp [8/256/0/182] [ 1118.399242] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1118.607780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1118.672631] LNet: 55186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.676406] LNetError: 55186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1118.680811] LNet: Removed LNI 10.1.2.3@tcp [ 1118.688909] LNetError: 55186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.748403] LNet: Removed LNI 192.168.201.59@tcp [ 1118.773498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1118.896481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1118.933486] LNet: Added LNI 192.168.201.59@tcp [8/256/0/182] [ 1118.937607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1118.939270] LNet: 55194:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1119.011418] LNet: Removed LNI 10.1.2.3@tcp [ 1119.019331] LNet: Removed LNI 192.168.201.59@tcp [ 1120.111943] Key type .llcrypt unregistered [ 1120.113937] Key type ._llcrypt unregistered [ 1132.761565] Key type ._llcrypt registered [ 1132.769380] Key type .llcrypt registered [ 1134.643441] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 19:52:27 (1790898747) [ 1136.206544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1138.981939] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:52:31 (1790898751) [ 1139.603484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1139.637558] alg: No test for adler32 (adler32-zlib) [ 1140.756188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1140.920395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1141.017180] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1142.415544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.59 [ 1142.442380] LNet: 56473:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1142.451690] LNet: Removed LNI 192.168.201.59@tcp [ 1151.048658] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:52:43 (1790898763) [ 1151.288251] LNet: 57047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.309538] LNetError: 57047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.355605] LNet: Removed LNI 192.168.201.59@tcp [ 1152.147244] Key type .llcrypt unregistered [ 1152.153876] Key type ._llcrypt unregistered [ 1160.284153] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1160.753515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1162.562936] Key type ._llcrypt registered [ 1162.567985] Key type .llcrypt registered [ 1162.774792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1162.801666] alg: No test for adler32 (adler32-zlib) [ 1164.397350] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1172.730261] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:53:05 (1790898785) [ 1172.874569] LNet: 57961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.885626] LNetError: 57961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1172.911435] LNet: Removed LNI 10.1.2.3@tcp [ 1173.544690] Key type .llcrypt unregistered [ 1173.551118] Key type ._llcrypt unregistered [ 1181.207771] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1181.536425] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1181.538537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1181.545259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1184.412573] Key type ._llcrypt registered [ 1184.417728] Key type .llcrypt registered [ 1184.615823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1184.636819] alg: No test for adler32 (adler32-zlib) [ 1185.620922] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1193.320732] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:53:25 (1790898805) [ 1195.440474] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1197.681774] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:53:30 (1790898810) [ 1197.760823] LNet: 59156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.770468] LNetError: 59156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1197.798987] LNet: Removed LNI 10.1.2.3@tcp [ 1198.695784] Key type .llcrypt unregistered [ 1198.711511] Key type ._llcrypt unregistered [ 1206.371121] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1206.756727] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1206.770509] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1206.783371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1209.713954] Key type ._llcrypt registered [ 1209.718409] Key type .llcrypt registered [ 1209.966531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1209.996420] alg: No test for adler32 (adler32-zlib) [ 1211.055965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1211.138034] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1220.960183] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:53:53 (1790898833) [ 1221.679679] LNet: 60406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.684473] LNetError: 60406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1221.710806] LNet: Removed LNI 10.1.2.3@tcp [ 1222.572584] Key type .llcrypt unregistered [ 1222.574209] Key type ._llcrypt unregistered [ 1230.569623] Key type ._llcrypt registered [ 1230.571470] Key type .llcrypt registered [ 1230.808865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1230.823322] alg: No test for adler32 (adler32-zlib) [ 1231.784516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1231.826121] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1242.168653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1243.612605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 1243.682607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 1243.780809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1243.838400] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1244.134316] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1244.220726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.284204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.333272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.380156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.450702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1244.506526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.563578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.645532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.729442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.795246] LNet: 60849:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.846581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.925539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.975236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.021188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.073709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.109738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.165262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1246.101688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 1246.159553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 1246.227149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1246.265960] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1246.272504] LNet: Skipped 1 previous similar message [ 1246.624639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.674694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1246.769285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.826454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.874590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1246.925593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.965199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.057356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.142263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.164125] LNet: 60960:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.199509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.256665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.310294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.354739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.399424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.435199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.488035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1248.316546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 1248.367634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 1248.436678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1248.484969] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1248.989658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.029579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1249.095565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.163772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.240530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1249.315532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.359333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.412673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.500572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.523392] LNet: 61072:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.570258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.625363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.708621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.789323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.861404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.911593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.965542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1250.676340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 1250.716915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 1250.753665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1250.774073] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1251.074320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.112620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1251.213597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.247208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.290765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1251.329223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1251.384724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1251.437448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.493638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.555275] LNet: 61183:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1251.596927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.667781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.725489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.784359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.824597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.870390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.908263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1252.631147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 1252.693822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 1252.748794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1253.033426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1253.061476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1253.087310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1253.137706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1253.170317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1253.199418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1253.230155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1253.265451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1253.306365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1253.349314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1253.381709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1253.415596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.454409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.490089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.544377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1253.638968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1254.308272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 1254.346388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 1254.379300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1254.628833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.677313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1254.731193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.776570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.848131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1254.904296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.946300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.983636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1255.040879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1255.080774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.131868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.177544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.220132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.267633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.325387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.407108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1255.797684] LNet: 61447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.807333] LNetError: 61447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.871177] LNet: Removed LNI 192.168.201.59@tcp [ 1257.574976] Key type .llcrypt unregistered [ 1257.577422] Key type ._llcrypt unregistered [ 1281.085746] Key type ._llcrypt registered [ 1281.087540] Key type .llcrypt registered [ 1283.204904] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:54:55 (1790898895) [ 1284.804564] Key type .llcrypt unregistered [ 1284.816927] Key type ._llcrypt unregistered [ 1292.662636] Key type ._llcrypt registered [ 1292.668219] Key type .llcrypt registered [ 1292.943910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1292.967905] alg: No test for adler32 (adler32-zlib) [ 1293.921118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1293.947519] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1302.710166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1314.797206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.903328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1314.943499] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1316.314951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1317.369533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 1317.419829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 1 [ 1317.457278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 1317.527212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 1 [ 1317.592446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 1317.633363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 1 [ 1317.676367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 1317.719850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 1 [ 1317.773631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1317.807202] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1317.814293] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp1 [ 1318.112962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1318.154465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1318.200222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1318.247299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1318.281475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1318.314144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1318.348747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1318.397152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1318.442804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1318.479244] LNet: 62938:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1318.511291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1318.550228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1318.591154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.639417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1318.674478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.725284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.778168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1319.369425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 1319.417820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 1 [ 1319.452498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 1319.499733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 1 [ 1319.535368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 1319.574501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 1 [ 1319.609334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 1319.649639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 1 [ 1319.693945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1319.717871] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1319.726811] LNet: Skipped 2 previous similar messages [ 1319.994630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1320.026796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1320.050503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.074988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.102755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1320.130620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.163034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1320.184359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1320.210917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1320.224053] LNet: 63061:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1320.239981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1320.265117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1320.299641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.331512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1320.367849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1320.402946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1320.446895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1320.974534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 1321.018588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 1 [ 1321.067414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 1321.123297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 1 [ 1321.156379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 1321.194463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 1 [ 1321.241993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 1321.293503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 1 [ 1321.343922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1321.365468] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1321.371467] LNet: Skipped 2 previous similar messages [ 1321.676592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1321.709865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1321.753698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1321.780816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1321.814849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1321.840718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1321.876796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1321.924313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1321.957320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1321.976815] LNet: 63185:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1322.001352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1322.036868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1322.072197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1322.096928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1322.125442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1322.161889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1322.199942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1322.753698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 1322.803920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 1 [ 1322.836908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 1322.864886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 1 [ 1322.892353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 1322.947586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 1 [ 1322.985452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 1323.042990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 1 [ 1323.083471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1323.426311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1323.462213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1323.508471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1323.533154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1323.582903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1323.621343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1323.658409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1323.685835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1323.722625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1323.771232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1323.826922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1323.856675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1323.886316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1323.916907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1323.946352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1323.974134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1324.453216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 1324.501382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 1 [ 1324.561820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 1324.606327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 1 [ 1324.653115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 1324.696431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 1 [ 1324.742950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 1324.779418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 1 [ 1324.844931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1324.886547] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1324.892096] LNet: Skipped 5 previous similar messages [ 1325.123696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1325.154753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1325.184221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1325.228802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1325.278981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1325.322440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1325.365686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1325.410234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1325.469607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1325.496847] LNet: 63431:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1325.502799] LNet: 63431:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1325.522822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1325.563391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1325.656901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1325.712675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1325.768897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1325.823686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1325.878240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1326.489975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 1326.514919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_error -r 1 [ 1326.551189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 1326.582897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_error -r 1 [ 1326.611832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 1326.643320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_error -r 1 [ 1326.672687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 1326.701740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_error -r 1 [ 1326.729920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1326.922344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1326.949799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1326.973575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1327.001825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1327.025767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1327.055660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1327.084856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1327.108728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1327.139433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1327.172284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1327.199147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1327.228456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1327.284994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1327.328323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1327.376186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1327.410803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1333.190525] LNet: 63631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.220566] LNetError: 63631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.237859] LNet: Removed LNI 192.168.201.59@tcp [ 1333.270294] LNet: Removed LNI 192.168.201.59@tcp1 [ 1333.904090] Key type .llcrypt unregistered [ 1333.915684] Key type ._llcrypt unregistered [ 1354.481764] Key type ._llcrypt registered [ 1354.484415] Key type .llcrypt registered [ 1355.986296] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:56:08 (1790898968) [ 1357.085335] Key type .llcrypt unregistered [ 1357.087389] Key type ._llcrypt unregistered [ 1364.321564] Key type ._llcrypt registered [ 1364.323619] Key type .llcrypt registered [ 1364.704221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1364.724976] alg: No test for adler32 (adler32-zlib) [ 1365.796691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1365.830219] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1375.040313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1376.047433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1376.102490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1376.145613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1376.168484] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1376.474764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1376.492191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1376.518686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1376.556273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1376.603870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1376.649096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1376.683590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1376.731735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1376.793314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1376.827853] LNet: 64899:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1376.853928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1376.906876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1376.965204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1377.021735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1377.081972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1377.176043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1377.235873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1378.000500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1378.061733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1378.149398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1378.188180] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1378.473652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1378.507558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1378.538797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1378.580663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1378.623132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1378.663639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1378.692155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1378.717959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1378.751878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1378.772687] LNet: 65010:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1378.799255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1378.842581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1378.893702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1378.962357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1379.006973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1379.051961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1379.081607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1379.823150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1379.858294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1379.934487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1379.955860] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1380.329607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1380.388257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1380.441336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1380.525931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1380.593251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1380.687419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1380.792728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1380.852637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1380.889780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1380.929179] LNet: 65121:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1380.995903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1381.061463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1381.144845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1381.229512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1381.287696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1381.344323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1381.428295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1381.841906] LNet: 65162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.852984] LNetError: 65162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.893485] LNet: Removed LNI 192.168.201.59@tcp [ 1383.583483] Key type .llcrypt unregistered [ 1383.589764] Key type ._llcrypt unregistered [ 1403.076865] Key type ._llcrypt registered [ 1403.078859] Key type .llcrypt registered [ 1405.231373] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:56:57 (1790899017) [ 1406.450063] Key type .llcrypt unregistered [ 1406.451766] Key type ._llcrypt unregistered [ 1414.363321] Key type ._llcrypt registered [ 1414.373534] Key type .llcrypt registered [ 1414.656638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1414.706331] alg: No test for adler32 (adler32-zlib) [ 1415.739581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1415.783593] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1424.938328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1435.972992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1436.076228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1436.133582] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1437.522659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1438.505824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1438.555507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 1 [ 1438.609550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1438.689826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 1 [ 1438.745461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1438.814420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 1 [ 1438.878934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1438.954345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 1 [ 1439.023764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1439.045983] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1439.060168] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp1 [ 1439.499404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1439.587897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1439.671491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1439.755228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1439.831276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1439.876457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1439.929862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1439.975276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1440.031724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1440.062742] LNet: 66653:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1440.096820] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1440.102284] LNet: Skipped 3 previous similar messages [ 1440.107337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1440.166497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1440.207674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1440.250150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1440.292691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1440.332037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1440.370397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1440.405753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1440.990540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1441.031425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_error -r 1 [ 1441.067347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1441.108376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_error -r 1 [ 1441.159486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1441.208730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_error -r 1 [ 1441.261413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1441.309301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_error -r 1 [ 1441.395358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1441.428064] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1441.712665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1441.750296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1441.829325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1441.918232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1441.967238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1442.033380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1442.074776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1442.102704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1442.137578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1442.170115] LNet: 66778:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1442.186878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1442.231965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1442.267746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1442.301834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1442.362561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1442.410728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1442.439880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1442.483290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1443.173540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1443.229204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_timeout -r 1 [ 1443.305109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1443.363603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_timeout -r 1 [ 1443.425441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1443.470315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_timeout -r 1 [ 1443.543271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1443.626335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_timeout -r 1 [ 1443.680077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1443.734982] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1444.084678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1444.128260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1444.186284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1444.233215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1444.284224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1444.346572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1444.391413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1444.432876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1444.463867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1444.498607] LNet: 66904:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1444.514527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1444.567551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1444.604455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1444.648375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1444.690973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1444.733736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1444.802223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1444.855124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1451.494465] LNet: 66982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.502236] LNetError: 66982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.512585] LNet: Removed LNI 192.168.201.59@tcp [ 1451.526678] LNet: Removed LNI 192.168.201.59@tcp1 [ 1451.887595] Key type .llcrypt unregistered [ 1451.892467] Key type ._llcrypt unregistered [ 1470.351344] Key type ._llcrypt registered [ 1470.354474] Key type .llcrypt registered [ 1471.794308] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:58:04 (1790899084) [ 1473.501607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1476.699073] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:58:08 (1790899088) [ 1477.952301] Key type .llcrypt unregistered [ 1477.955891] Key type ._llcrypt unregistered [ 1485.973808] Key type ._llcrypt registered [ 1485.975147] Key type .llcrypt registered [ 1486.492833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1486.517261] alg: No test for adler32 (adler32-zlib) [ 1487.733804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1487.782644] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1497.159250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1498.069318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1498.121683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1498.169689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1498.203230] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1498.497224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1498.538996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1498.583506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1498.622438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1498.659579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1498.725434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1498.785192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1498.847862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1498.923902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1498.954989] LNet: 68558:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1498.991402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1499.060228] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1499.067225] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1499.090410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1499.153293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1499.221400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1499.312267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1499.402436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1499.874813] LNet: 68598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.896908] LNetError: 68598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.936262] LNet: Removed LNI 192.168.201.59@tcp [ 1500.715439] Key type .llcrypt unregistered [ 1500.720122] Key type ._llcrypt unregistered [ 1525.715521] Key type ._llcrypt registered [ 1525.720489] Key type .llcrypt registered [ 1526.097494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1526.120381] alg: No test for adler32 (adler32-zlib) [ 1527.093906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1527.121276] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1535.793507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1546.160767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.241206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1546.275043] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1547.200782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1547.865313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1547.900029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e network_timeout -r 1 [ 1547.982549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1548.056242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e network_timeout -r 1 [ 1548.126526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1548.183500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e network_timeout -r 1 [ 1548.280241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1548.335234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e network_timeout -r 1 [ 1548.390404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1548.420778] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1548.639094] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp1 [ 1548.642930] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp1 [ 1548.707548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1548.751880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1548.786746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1548.835863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1548.871826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1548.940745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1548.991838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1549.058870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1549.097607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1549.134219] LNet: 69493:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1549.166569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1549.197545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1549.234867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1549.266805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1549.293850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1549.341514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1555.289671] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1555.835827] LNet: 69567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.867525] LNetError: 69567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.873475] LNet: Removed LNI 192.168.201.59@tcp [ 1555.886128] LNet: Removed LNI 192.168.201.59@tcp1 [ 1556.514575] Key type .llcrypt unregistered [ 1556.517275] Key type ._llcrypt unregistered [ 1582.538693] Key type ._llcrypt registered [ 1582.540778] Key type .llcrypt registered [ 1584.136703] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:59:56 (1790899196) [ 1584.683183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1584.702482] alg: No test for adler32 (adler32-zlib) [ 1585.585624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.649265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1585.681675] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1586.595335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1586.627393] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1587.954887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1588.063608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1588.144638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1588.211503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1588.329166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1588.375267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1588.422662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1588.449760] LNetError: 70680:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1588.703351] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1588.706567] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1590.761562] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1590.960420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1594.849932] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1595.820113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1595.890556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1596.011425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.081606] LNet: 70787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.091628] LNetError: 70787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.097798] LNet: Removed LNI 192.168.201.59@tcp [ 1596.116852] LNet: Removed LNI 192.168.201.59@tcp1 [ 1596.136627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.213859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.248896] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1597.283996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1597.315355] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1598.302563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1598.387497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1598.444199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1598.489368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1598.551605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1598.608401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1598.678653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1598.713703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1598.733412] LNetError: 71060:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1599.264314] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1602.237858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1605.410465] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1605.428574] LNet: Skipped 1 previous similar message [ 1610.363588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1610.401803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1610.456707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1619.007453] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:00:31 (1790899231) [ 1619.158446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.210681] LNet: 71780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.220188] LNetError: 71780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.233568] LNetError: 71780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1619.241091] LNet: Removed LNI 192.168.201.59@tcp [ 1619.264432] LNet: Removed LNI 192.168.201.59@tcp1 [ 1619.368899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.451435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1619.482965] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1620.361132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1620.383845] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1621.089292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1621.152417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1621.180741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1621.213164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1621.252691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1621.289517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1621.323819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 0 [ 1621.408714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1621.472829] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.59@tcp [ 1622.498923] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1622.515707] LNet: Skipped 1 previous similar message [ 1637.209217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1637.518287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1637.608971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1637.672774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 500 [ 1637.776309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1644.581872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1644.628377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.648391] LNet: 72279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1644.653394] LNetError: 72279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1644.663387] LNetError: 72279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1644.669528] LNet: Removed LNI 192.168.201.59@tcp [ 1644.678943] LNet: Removed LNI 192.168.201.59@tcp1 [ 1644.698389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.754741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.787267] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1645.748414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1645.784356] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1646.980313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1647.018512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1647.167833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1647.286657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1647.395311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1647.458214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1647.546947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1647.614576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 0 [ 1648.799364] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1648.804888] LNet: Skipped 3 previous similar messages [ 1649.084452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1658.936440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1659.042782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1659.166809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1669.794769] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:01:22 (1790899282) [ 1670.349410] LNet: 73338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.364894] LNetError: 73338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.372376] LNetError: 73338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1671.391859] LNet: Removed LNI 192.168.201.59@tcp [ 1671.402272] LNet: Removed LNI 192.168.201.59@tcp1 [ 1672.066824] Key type .llcrypt unregistered [ 1672.069535] Key type ._llcrypt unregistered [ 1680.500123] Key type ._llcrypt registered [ 1680.503326] Key type .llcrypt registered [ 1680.916976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1680.956619] alg: No test for adler32 (adler32-zlib) [ 1682.032274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1682.106185] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1693.076110] Lustre: DEBUG MARKER: Initial discovery [ 1693.309399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1695.882418] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1696.009983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1696.087993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1696.157969] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1696.202851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1696.258768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.159@tcp -r 1 -m GET -l 3 [ 1696.307621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.159@tcp -r 1 -m GET -e local_error [ 1696.375914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.159@tcp -r 1 -m PUT -l 6 [ 1696.479704] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1698.116561] Lustre: DEBUG MARKER: Force oleg159-client.virtnet to discover 192.168.201.159@tcp (in background) [ 1698.337655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1699.967773] Lustre: DEBUG MARKER: Force oleg159-server to discover 192.168.201.59@tcp [ 1701.420230] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1702.883343] Lustre: DEBUG MARKER: Wait for 74110 [ 1704.798767] Lustre: DEBUG MARKER: Finished wait on 74110 [ 1705.079073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1705.111493] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1705.123634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1705.422081] LNet: 74565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.427813] LNetError: 74565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.446261] LNet: Removed LNI 192.168.201.59@tcp [ 1705.835510] Key type .llcrypt unregistered [ 1705.844843] Key type ._llcrypt unregistered [ 1725.693181] Key type ._llcrypt registered [ 1725.694971] Key type .llcrypt registered [ 1726.862367] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:02:20 (1790899340) [ 1727.468352] Key type .llcrypt unregistered [ 1727.475289] Key type ._llcrypt unregistered [ 1734.824254] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1735.090148] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1735.094358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1735.098916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1737.910760] Key type ._llcrypt registered [ 1737.917163] Key type .llcrypt registered [ 1738.157741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1738.196820] alg: No test for adler32 (adler32-zlib) [ 1739.282878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.380646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.439805] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1740.553405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1740.586429] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1752.434771] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:02:45 (1790899365) [ 1752.889775] LNet: 76668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.909131] LNetError: 76668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1752.915694] LNet: Removed LNI 10.1.2.3@tcp [ 1752.935316] LNet: Removed LNI 192.168.201.59@tcp [ 1753.657759] Key type .llcrypt unregistered [ 1753.659452] Key type ._llcrypt unregistered [ 1762.439327] Key type ._llcrypt registered [ 1762.441898] Key type .llcrypt registered [ 1762.620775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1762.652591] alg: No test for adler32 (adler32-zlib) [ 1763.704198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.792514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.855831] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1764.896372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1764.913424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1766.002944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.59@tcp [ 1767.071250] LNetError: Unexpected error -22 connecting to 192.168.201.59@tcp at host 192.168.201.59:988 [ 1774.019765] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:03:06 (1790899386) [ 1774.229386] LNet: 77804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.260927] LNetError: 77804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1774.291381] LNet: Removed LNI 10.1.2.3@tcp [ 1774.318317] LNet: Removed LNI 192.168.201.59@tcp [ 1775.757959] Key type .llcrypt unregistered [ 1775.767756] Key type ._llcrypt unregistered [ 1782.271268] Key type ._llcrypt registered [ 1782.273985] Key type .llcrypt registered [ 1782.407928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1782.423871] alg: No test for adler32 (adler32-zlib) [ 1783.327776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.372646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1783.395787] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1784.090181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1784.114802] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1785.237613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.59@tcp1 --nid 192.168.201.59@tcp2 [ 1790.078133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp1 192.168.201.59@tcp2 [ 1790.124633] LNetError: 78648:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.59@tcp1 to 192.168.201.59@tcp2 - Route table may be misconfigured [ 1790.177696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp2 192.168.201.59@tcp1 [ 1797.205683] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:03:30 (1790899410) [ 1797.594072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.638416] LNet: 79231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.648223] LNetError: 79231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.675124] LNet: Removed LNI 192.168.201.59@tcp1 [ 1798.694357] LNet: Removed LNI 192.168.201.59@tcp2 [ 1798.721891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.793386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.818493] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1799.455379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1799.490764] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1800.375795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1800.407206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.59@tcp [ 1800.433623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp1 192.168.201.59@tcp1 [ 1800.464418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.59@tcp -d 192.168.201.59@tcp -e network_timeout [ 1800.490694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -e network_timeout [ 1800.524464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -e network_timeout [ 1800.564559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -e network_timeout [ 1800.602979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1800.620517] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1800.633457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1800.721262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1800.799706] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1800.811710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1807.584476] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:03:40 (1790899420) [ 1807.801610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.853163] LNet: 80099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.873704] LNetError: 80099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.880223] LNetError: 80099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1807.888234] LNet: Removed LNI 192.168.201.59@tcp [ 1808.933925] LNet: Removed LNI 192.168.201.59@tcp1 [ 1808.995520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.070968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1809.379468] LNet: 80138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.917430] Key type .llcrypt unregistered [ 1809.919188] Key type ._llcrypt unregistered [ 1823.211746] Key type ._llcrypt registered [ 1823.216522] Key type .llcrypt registered [ 1825.420457] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:03:57 (1790899437) [ 1825.989790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.041892] alg: No test for adler32 (adler32-zlib) [ 1826.976275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.075645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1831.705508] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:04:03 (1790899443) [ 1831.853567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.895393] LNet: 81242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.941943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.079961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1832.108815] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1833.235633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1833.293857] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1834.491370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1834.530122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp1 [ 1834.567522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp1 [ 1843.997513] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:04:16 (1790899456) [ 1845.977873] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg159-server [ 1848.623973] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:04:20 (1790899460) [ 1852.225909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg159-server [ 1855.193438] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:04:27 (1790899467) [ 1858.740714] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg159-server [ 1860.922736] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:04:33 (1790899473) [ 1863.900851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg159-server [ 1866.775513] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:04:39 (1790899479) [ 1870.103107] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg159-server [ 1872.731753] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:04:45 (1790899485) [ 1875.258310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg159-server [ 1877.225841] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:04:50 (1790899490) [ 1879.388324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg159-server [ 1881.031316] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:04:54 (1790899494) [ 1883.395677] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg159-server [ 1886.218877] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:04:58 (1790899498) [ 1889.342735] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg159-server [ 1891.610957] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:05:04 (1790899504) [ 1891.827617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1891.845811] LNet: 85191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.852989] LNetError: 85191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1891.860768] LNet: Removed LNI 192.168.201.59@tcp [ 1892.901489] LNet: Removed LNI 192.168.201.59@tcp1 [ 1892.962740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.054203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1893.083491] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1893.855714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1893.920934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1895.206508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.226762] LNet: 85347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.231263] LNetError: 85347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.236050] LNetError: 85347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1895.252819] LNet: Removed LNI 192.168.201.59@tcp [ 1895.269557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.320648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1895.340858] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1896.085334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1896.243222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.266042] LNetError: 85493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.292377] LNet: Removed LNI 192.168.201.59@tcp [ 1896.346991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.412347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1896.431932] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1897.058889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1897.115907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1897.381862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1897.410474] LNet: 85640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.418314] LNet: 85640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1898.488784] LNet: Removed LNI 192.168.201.59@tcp [ 1898.558965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.640403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1898.682794] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1899.598495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1899.752357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1899.768618] LNetError: 85785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.773583] LNetError: 85785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1900.842969] LNet: Removed LNI 192.168.201.59@tcp [ 1900.889511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.960424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1901.002445] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1901.713191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1901.794761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1902.019600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1902.051671] LNet: 85932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.058478] LNet: 85932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1903.149455] LNet: Removed LNI 192.168.201.59@tcp [ 1903.206231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.284822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1903.322862] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1904.414248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1905.780149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.803358] LNetError: 86087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.812134] LNetError: 86087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1905.831926] LNet: Removed LNI 192.168.201.59@tcp [ 1905.863547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.935570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1906.001329] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1906.653531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1906.826334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1907.134006] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.200638] LNet: Removed LNI 192.168.201.59@tcp [ 1908.255371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.354521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1908.395512] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1909.603825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1909.935833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.016420] LNet: Removed LNI 192.168.201.59@tcp [ 1911.079210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.185391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1911.228341] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1911.946909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1912.029251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1912.271282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.306272] LNet: 86527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.312242] LNet: 86527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1913.403713] LNet: Removed LNI 192.168.201.59@tcp [ 1913.434836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.567183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1913.608321] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1914.558412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1914.949818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.986139] LNetError: 86672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.989630] LNetError: 86672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1916.014425] LNet: Removed LNI 192.168.201.59@tcp [ 1916.102243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.190733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1916.232478] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1917.047475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1917.150623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1917.437099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1918.506936] LNet: Removed LNI 192.168.201.59@tcp [ 1918.545908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.623953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1918.662622] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1919.430406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1919.815965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.874860] LNet: Removed LNI 192.168.201.59@tcp [ 1920.906830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.956877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1920.973152] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1921.732691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1921.785403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.810088] LNet: Removed LNI 192.168.201.59@tcp [ 1921.833711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.923435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1921.944771] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1922.924496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1922.995691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.017533] LNet: Removed LNI 192.168.201.59@tcp [ 1923.043238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.096591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1923.174549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.227720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.298657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1923.404919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.453956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.541518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1923.705787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.775440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.850850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1923.910096] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1924.867566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1924.907927] LNet: Removed LNI 192.168.201.59@tcp [ 1924.938585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.024617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1932.533282] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:05:45 (1790899545) [ 1932.700936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.736667] LNet: 87989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.751713] LNet: 87989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1932.790747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.874716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1932.924847] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1934.317598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1934.329697] LNetError: 88132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1934.333794] LNetError: 88132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1934.343316] LNet: Removed LNI 192.168.201.59@tcp231 [ 1934.359596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1934.481084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1934.515555] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1934.521543] LNet: 88140:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1935.111628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1935.146491] LNet: Removed LNI 192.168.201.59@tcp231 [ 1935.162842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1935.193040] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1936.522247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.559464] LNet: Removed LNI 192.168.201.59@tcp231 [ 1936.573860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.638437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1936.661883] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/0] [ 1936.667283] LNet: 88292:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1937.165430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1937.209695] LNet: Removed LNI 192.168.201.59@tcp231 [ 1937.228852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1937.263739] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/0] [ 1938.536107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1938.584686] LNet: Removed LNI 192.168.201.59@tcp231 [ 1938.622660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.723767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1938.775153] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/60] [ 1938.781447] LNet: 88444:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1939.243302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1939.275075] LNet: Removed LNI 192.168.201.59@tcp231 [ 1939.309088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1939.355859] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/60] [ 1940.738481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1940.779906] LNet: Removed LNI 192.168.201.59@tcp231 [ 1940.802178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1940.888628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1940.918754] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1940.923946] LNet: 88595:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1941.314388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1941.347749] LNet: Removed LNI 192.168.201.59@tcp231 [ 1941.374733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1941.405738] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1950.997473] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:06:03 (1790899563) [ 1951.057807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1951.123355] LNet: Removed LNI 192.168.201.59@tcp231 [ 1951.152196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1951.298235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1951.340139] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1952.380303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1952.451247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1952.483272] LNet: Removed LNI 192.168.201.59@tcp232 [ 1952.512721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1952.539444] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1953.465779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1953.506984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1953.536242] LNet: Removed LNI 192.168.201.59@tcp232 [ 1953.559198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1953.583932] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1954.385061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1954.430307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1954.470812] LNet: Removed LNI 192.168.201.59@tcp232 [ 1954.509846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1954.541917] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1955.238834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1955.279567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1955.314245] LNet: Removed LNI 192.168.201.59@tcp232 [ 1955.331493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1962.548754] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:06:15 (1790899575) [ 1963.735600] Key type .llcrypt unregistered [ 1963.743548] Key type ._llcrypt unregistered [ 1971.290781] Key type ._llcrypt registered [ 1971.292597] Key type .llcrypt registered [ 1971.505425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1971.513572] alg: No test for adler32 (adler32-zlib) [ 1972.436373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1972.465777] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1980.233853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1989.212771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.243827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1989.255702] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1990.469418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1991.131832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 2 [ 1991.165351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 2 [ 1991.201627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 2 [ 1991.222590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 2 [ 1991.248198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_interrupt -r 2 [ 1991.264725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 2 [ 1991.282844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_interrupt -r 2 [ 1991.303733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 2 [ 1991.328530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1991.351838] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1991.605288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1991.627913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1991.659884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1991.690544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1991.716842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1991.750542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1991.776778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1991.805780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1991.835295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1991.853521] LNet: 91122:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1991.872305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1991.904635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1991.933250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1991.960377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1991.991370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1992.019906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1992.577418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 2 [ 1992.624812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 2 [ 1992.646480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 2 [ 1992.675928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 2 [ 1992.702290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_dropped -r 2 [ 1992.728709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 2 [ 1992.754510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_dropped -r 2 [ 1992.781757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 2 [ 1992.827876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1992.855845] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1992.864924] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1993.194918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1993.230043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1993.276673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1993.333496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1993.385368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1993.431858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1993.489473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1993.534585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1993.580811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1993.605915] LNet: 91244:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1993.634586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1993.678412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1993.727545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1993.752810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1993.797823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1993.869848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1994.623082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 2 [ 1994.674884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 2 [ 1994.712740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 2 [ 1994.767684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 2 [ 1994.820395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_aborted -r 2 [ 1994.900303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 2 [ 1994.977979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_aborted -r 2 [ 1995.044877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 2 [ 1995.095453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1995.141258] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1995.520449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1995.549785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1995.578614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1995.610811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1995.644173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1995.678264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1995.707960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1995.741871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1995.781435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1995.802182] LNet: 91367:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1995.822306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1995.853561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1995.890723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1995.941670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1996.018763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1996.101306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1996.703943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 2 [ 1996.736958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 2 [ 1996.771369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 2 [ 1996.801278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 2 [ 1996.847420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_no_route -r 2 [ 1996.871797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 2 [ 1996.893555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_no_route -r 2 [ 1996.913048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 2 [ 1996.930480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1997.147767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1997.192038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1997.218594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1997.257714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1997.316961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1997.354454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1997.400941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1997.459649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1997.501795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1997.539602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1997.567417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1997.609443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1997.639222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1997.666540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1997.691764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1998.396395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 2 [ 1998.442694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 2 [ 1998.513771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 2 [ 1998.553757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 2 [ 1998.605281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_timeout -r 2 [ 1998.662243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 2 [ 1998.746936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_timeout -r 2 [ 1998.860603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 2 [ 1998.911913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1998.936361] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1998.943601] LNet: Skipped 1 previous similar message [ 1999.270988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1999.300853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1999.323777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1999.350451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1999.378457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1999.420361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1999.453868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1999.500756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1999.530360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1999.550718] LNet: 91612:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1999.558113] LNet: 91612:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1999.586912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1999.627772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1999.660671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1999.692383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1999.742273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1999.794342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2000.462815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 2 [ 2000.493574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 2 [ 2000.528498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 2 [ 2000.562606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 2 [ 2000.618298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_dropped -r 2 [ 2000.658479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 2 [ 2000.698924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_dropped -r 2 [ 2000.728995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 2 [ 2000.773584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 2000.798505] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 2001.019867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2001.055549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2001.083321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2001.109763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2001.142090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2001.173632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2001.195362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2001.223769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2001.247213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2001.274583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2001.299814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2001.326587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2001.355088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2001.392829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2001.422686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2009.106671] LNet: 91809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.111909] LNetError: 91809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.119174] LNet: Removed LNI 192.168.201.59@tcp [ 2009.134071] LNet: Removed LNI 192.168.201.59@tcp1 [ 2009.638745] Key type .llcrypt unregistered [ 2009.648236] Key type ._llcrypt unregistered [ 2031.166070] Key type ._llcrypt registered [ 2031.168307] Key type .llcrypt registered [ 2033.024702] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:07:25 (1790899645) [ 2033.675130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2033.688418] alg: No test for adler32 (adler32-zlib) [ 2034.800071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.909313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2034.936091] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2036.094182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2036.150303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 2036.173683] LNetError: 92795:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 2036.301455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.59@tcp [ 2036.424962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2043.828764] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:07:36 (1790899656) [ 2044.247074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2044.280152] LNet: 93394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.290852] LNetError: 93394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2044.312049] LNet: Removed LNI 192.168.201.59@tcp [ 2044.359376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2044.500441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2044.530716] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2045.464097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2045.506340] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 2046.512898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.59@tcp [ 2046.528212] LNetError: 93660:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 2047.519655] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 2047.521901] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 2047.842731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.59@tcp [ 2057.386216] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:07:50 (1790899670) [ 2059.918783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg159-server [ 2063.335629] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 20:07:55 (1790899675) [ 2066.421946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg159-server [ 2069.580110] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 20:08:01 (1790899681) [ 2072.630794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg159-server [ 2075.638049] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:08:07 (1790899687) [ 2075.874656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2075.927635] LNet: 95315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.940395] LNetError: 95315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.953218] LNet: Removed LNI 192.168.201.59@tcp [ 2075.975123] LNet: Removed LNI 192.168.201.59@tcp1 [ 2076.058758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.169127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2076.246263] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2077.637290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2077.809825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2077.870740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2078.099468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2078.140954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2078.263208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2085.855218] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:08:18 (1790899698) [ 2086.111188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2086.120585] LNet: 96058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.123110] LNetError: 96058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.125288] LNetError: 96058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2086.140038] LNet: Removed LNI 192.168.201.59@tcp [ 2086.154471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.235293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2086.253392] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2095.242071] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:08:28 (1790899708) [ 2095.442761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2095.521292] LNet: 96766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2095.526608] LNetError: 96766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2095.552638] LNet: Removed LNI 192.168.201.59@tcp [ 2095.603432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.756920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2095.797163] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2104.922156] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:08:37 (1790899717) [ 2107.089574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2109.313683] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:08:42 (1790899722) [ 2109.848761] LNet: 97820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.854809] LNetError: 97820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.871561] LNet: Removed LNI 192.168.201.59@tcp [ 2110.460991] Key type .llcrypt unregistered [ 2110.464719] Key type ._llcrypt unregistered [ 2118.143119] Key type ._llcrypt registered [ 2118.144666] Key type .llcrypt registered [ 2118.520945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2118.544095] alg: No test for adler32 (adler32-zlib) [ 2119.540230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2119.571173] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2127.336106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2133.250409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.159@tcp [ 2133.621368] LNet: 98239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.630521] LNetError: 98239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.662173] LNet: Removed LNI 192.168.201.59@tcp [ 2134.283637] Key type .llcrypt unregistered [ 2134.291502] Key type ._llcrypt unregistered [ 2148.746931] Key type ._llcrypt registered [ 2148.750113] Key type .llcrypt registered [ 2150.351573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2152.185930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2154.415503] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:09:27 (1790899767) [ 2156.936396] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2159.557651] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:09:32 (1790899772) [ 2163.000544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg159-server [ 2165.294734] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:09:38 (1790899778) [ 2168.166651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg159-server [ 2170.032563] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:09:43 (1790899783) [ 2171.793843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2171.936318] alg: No test for adler32 (adler32-zlib) [ 2172.950392] Lustre: Lustre: Build Version: 2.17.59_2_g8e10c0d [ 2173.181709] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2174.791235] Key type lgssc registered [ 2176.119584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2196.336261] Key type lgssc unregistered [ 2196.541492] LNet: 101493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.548316] LNetError: 101493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2196.560717] LNet: Removed LNI 192.168.201.59@tcp [ 2197.299585] Key type .llcrypt unregistered [ 2197.301210] Key type ._llcrypt unregistered [ 2212.033877] Key type ._llcrypt registered [ 2212.037619] Key type .llcrypt registered [ 2213.355301] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:10:26 (1790899826) [ 2213.659910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2213.677136] alg: No test for adler32 (adler32-zlib) [ 2214.564583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.615813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2214.652719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2214.689198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2214.735920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2221.455320] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:10:34 (1790899834) [ 2221.513915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.527128] LNet: 102926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.550369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.593970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2221.614980] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2222.329443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.343737] LNet: 103076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.347251] LNetError: 103076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.357898] LNet: Removed LNI 192.168.201.59@tcp [ 2222.371367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.408127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2222.436578] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2222.792957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.841072] LNet: Removed LNI 192.168.201.59@tcp [ 2222.891737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.927328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2222.943519] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2230.961740] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:10:44 (1790899844) [ 2231.726210] LNet: 103702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.728251] LNet: 103702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2231.749699] LNetError: 103702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.760908] LNetError: 103702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2231.790211] LNet: Removed LNI 192.168.201.59@tcp [ 2233.109883] Key type .llcrypt unregistered [ 2233.116711] Key type ._llcrypt unregistered [ 2239.269884] Key type ._llcrypt registered [ 2239.272452] Key type .llcrypt registered [ 2239.658578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2239.699455] alg: No test for adler32 (adler32-zlib) [ 2240.791741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.945697] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2240.951034] LNet: *** cfs_fail_loc=e003, val=100*** [ 2242.411861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2242.452510] LNet: *** cfs_fail_loc=e003, val=100*** [ 2242.459589] LNet: Skipped 2 previous similar messages [ 2242.472559] Ping reply received with mapping: [ 2242.481474] found metadata reply NI[192.168.201.59@tcp] [ 2242.484801] 3b c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2242.493218] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2242.498791] 5a 5a 5a 5a 5a 5a 5a 5a [ 2242.844966] LNet: 104102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.874915] LNetError: 104102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2243.958737] LNet: Removed LNI 192.168.201.59@tcp [ 2244.888995] Key type .llcrypt unregistered [ 2244.895189] Key type ._llcrypt unregistered [ 2260.720297] Key type ._llcrypt registered [ 2260.723648] Key type .llcrypt registered [ 2262.290179] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:11:15 (1790899875) [ 2263.402190] Key type .llcrypt unregistered [ 2263.404932] Key type ._llcrypt unregistered [ 2271.438490] Key type ._llcrypt registered [ 2271.441993] Key type .llcrypt registered [ 2271.825741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2271.845424] alg: No test for adler32 (adler32-zlib) [ 2272.980473] LNetError: 105157:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2273.571955] Key type .llcrypt unregistered [ 2273.574950] Key type ._llcrypt unregistered [ 2282.036850] Key type ._llcrypt registered [ 2282.038955] Key type .llcrypt registered [ 2284.267406] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:11:36 (1790899896) [ 2285.190394] Key type .llcrypt unregistered [ 2285.193518] Key type ._llcrypt unregistered [ 2292.768683] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2292.775822] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2295.682783] Key type ._llcrypt registered [ 2295.686071] Key type .llcrypt registered [ 2295.889209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2295.925601] alg: No test for adler32 (adler32-zlib) [ 2296.936544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.037389] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2309.386485] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:12:02 (1790899922) [ 2309.506318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.552271] LNet: 106952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.557477] LNetError: 106952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2309.578457] LNet: Removed LNI 10.1.2.3@tcp [ 2309.616916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.450484] Key type .llcrypt unregistered [ 2310.456615] Key type ._llcrypt unregistered [ 2318.951808] Key type ._llcrypt registered [ 2318.954901] Key type .llcrypt registered [ 2319.345572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2319.384493] alg: No test for adler32 (adler32-zlib) [ 2320.643902] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2320.767570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2320.977726] LNet: 107247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.982659] LNetError: 107247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2322.036055] LNet: Removed LNI 192.168.201.59@tcp [ 2323.601620] Key type .llcrypt unregistered [ 2323.603466] Key type ._llcrypt unregistered [ 2337.345458] Key type ._llcrypt registered [ 2337.347313] Key type .llcrypt registered [ 2339.216814] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:12:31 (1790899951) [ 2340.493794] Key type .llcrypt unregistered [ 2340.498734] Key type ._llcrypt unregistered [ 2350.578734] Key type ._llcrypt registered [ 2350.582243] Key type .llcrypt registered [ 2350.789222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2350.800227] alg: No test for adler32 (adler32-zlib) [ 2359.086052] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:12:52 (1790899972) [ 2359.499852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.539794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.612967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2359.634379] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2360.305542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.341789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2360.392378] LNet: Removed LNI 192.168.201.59@tcp [ 2360.669391] LNet: 109013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.129455] Key type .llcrypt unregistered [ 2361.134047] Key type ._llcrypt unregistered [ 2373.682317] Key type ._llcrypt registered [ 2373.684887] Key type .llcrypt registered [ 2375.073070] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:13:08 (1790899988) [ 2375.578807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2375.621149] alg: No test for adler32 (adler32-zlib) [ 2376.580947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2376.620848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2376.643346] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2377.493643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2377.538165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.59@tcp [ 2377.560143] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2377.562533] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2377.567183] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2377, 0, 0 [ 2377.571957] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2377.575225] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2377.581207] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2377.586291] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2377, 0, 0 [ 2377.596910] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2377.601709] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2377.606463] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2377.610304] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2377, 0, 2427 [ 2377.614660] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2377.616783] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2377.620888] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2377.624306] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2377, 0, 2427 [ 2377.629219] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2385.163546] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:13:18 (1790899998) [ 2386.109548] LNet: 110575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.120639] LNetError: 110575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2386.148850] LNet: Removed LNI 192.168.201.59@tcp [ 2387.042707] Key type .llcrypt unregistered [ 2387.047088] Key type ._llcrypt unregistered [ 2394.966736] Key type ._llcrypt registered [ 2394.969322] Key type .llcrypt registered [ 2395.141432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2395.153559] alg: No test for adler32 (adler32-zlib) [ 2396.107720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2396.129591] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2404.335965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2414.034829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2414.146496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2414.229316] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 2415.369362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2415.607834] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2415.849621] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2415.857581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2415.867983] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2417.246785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2417.277160] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2429.622364] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:14:02 (1790900042) [ 2429.825106] LNet: 112088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.833439] LNetError: 112088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.843417] LNet: Removed LNI 192.168.201.59@tcp [ 2430.883225] LNet: Removed LNI 192.168.201.59@tcp1 [ 2430.885154] LNetError: 112088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2430.889435] LNetError: 112088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2430.904387] LNet: Removed LNI 10.1.2.3@tcp99 [ 2431.773804] Key type .llcrypt unregistered [ 2431.778722] Key type ._llcrypt unregistered [ 2439.043564] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2439.051195] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2441.731517] Key type ._llcrypt registered [ 2441.736937] Key type .llcrypt registered [ 2441.962642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2441.975719] alg: No test for adler32 (adler32-zlib) [ 2442.931744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2443.024103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2443.067460] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2443.759743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2443.789454] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2446.543297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2446.625328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2446.691836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2446.727974] LNetError: 112833:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2446.795124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2446.903502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2447.288329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2447.383408] LNet: 112854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.389933] LNetError: 112854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2447.398877] LNet: Removed LNI 10.1.2.3@tcp [ 2448.423053] LNet: Removed LNI 192.168.201.59@tcp [ 2449.301548] Key type .llcrypt unregistered [ 2449.310985] Key type ._llcrypt unregistered [ 2462.046788] Key type ._llcrypt registered [ 2462.049333] Key type .llcrypt registered [ 2463.717745] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:14:36 (1790900076) [ 2464.113279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2464.134994] alg: No test for adler32 (adler32-zlib) [ 2465.136959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2465.196735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2465.239137] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2466.372625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2466.478725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg159-client@tcp [ 2473.132735] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:14:46 (1790900086) [ 2473.454492] LNet: 114404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.462262] LNetError: 114404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.494865] LNet: Removed LNI 192.168.201.59@tcp [ 2474.151721] Key type .llcrypt unregistered [ 2474.153969] Key type ._llcrypt unregistered [ 2482.180540] Key type ._llcrypt registered [ 2482.186750] Key type .llcrypt registered [ 2482.525445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2482.548337] alg: No test for adler32 (adler32-zlib) [ 2483.572262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2483.616885] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2483.687258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2491.332244] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:15:04 (1790900104) [ 2493.373884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2495.801448] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:15:08 (1790900108) [ 2497.909701] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2499.926853] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:15:13 (1790900113) [ 2500.406579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2500.478546] LNet: 115845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2500.487618] LNetError: 115845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2500.536270] LNet: Removed LNI 192.168.201.59@tcp [ 2500.569437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2500.672579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2500.722129] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2501.744338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2501.895320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2510.640422] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:15:23 (1790900123) [ 2510.820143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2510.863152] LNet: 116557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.883741] LNetError: 116557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.923611] LNet: Removed LNI 192.168.201.59@tcp [ 2510.971750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2511.058981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2511.100899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2511.743888] Key type .llcrypt unregistered [ 2511.747632] Key type ._llcrypt unregistered [ 2525.192889] Key type ._llcrypt registered [ 2525.195565] Key type .llcrypt registered [ 2526.859524] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:15:39 (1790900139) [ 2527.353850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2527.373769] alg: No test for adler32 (adler32-zlib) [ 2528.361899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2528.442689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2528.478809] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2529.665952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2529.751354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2536.572748] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:15:49 (1790900149) [ 2536.914529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2536.937162] LNet: 118137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.944504] LNetError: 118137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2536.987483] LNet: Removed LNI 192.168.201.59@tcp [ 2537.044267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2537.155665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2537.210796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2545.028296] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:15:57 (1790900157) [ 2545.183939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2545.246347] LNet: 118722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2545.291948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2545.484726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2546.151750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2546.190257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2546.243325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2546.759257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2546.822463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2547.018995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2547.062763] LNet: 118759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2547.129738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2547.298685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2547.783137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2547.845173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2547.916291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2548.925729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2548.983640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2557.280285] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:16:09 (1790900169) [ 2557.417712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2557.449764] LNet: 119363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2557.526487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2557.673631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2557.702653] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2558.788211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2558.824375] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2560.054672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2560.128268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2560.416545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2560.455478] LNetError: 119648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2560.462789] LNet: Removed LNI 192.168.201.59@tcp [ 2560.476826] LNet: Removed LNI 192.168.201.59@tcp2 [ 2560.504995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2560.582145] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2560.590595] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2560.598378] LNet: 119654:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2561.440586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2561.790447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2561.821588] LNet: 119691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2561.825905] LNet: 119691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2561.833902] LNetError: 119691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2561.842284] LNetError: 119691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2561.850088] LNet: Removed LNI 192.168.201.59@tcp [ 2561.871144] LNet: Removed LNI 192.168.201.59@tcp2 [ 2561.894131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2562.016443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2562.062130] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2563.067153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2563.089608] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2564.462956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2564.537614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2564.638789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2565.009908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2565.031344] LNetError: 119976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2565.041069] LNetError: 119976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2565.051206] LNet: Removed LNI 192.168.201.59@tcp [ 2566.121978] LNet: Removed LNI 192.168.201.59@tcp2 [ 2566.169655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2566.302198] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2566.313228] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2566.327985] LNet: 119982:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2567.262511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2567.572136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2567.630843] LNet: Removed LNI 192.168.201.59@tcp [ 2568.679623] LNet: Removed LNI 192.168.201.59@tcp2 [ 2568.730942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2568.809949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2568.855338] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2570.112485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2570.159453] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2571.356396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2571.460041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2571.782428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2571.823453] LNet: 120299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2571.831661] LNet: 120299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2571.848179] LNetError: 120299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2571.856163] LNetError: 120299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2571.869609] LNet: Removed LNI 192.168.201.59@tcp [ 2571.887303] LNet: Removed LNI 192.168.201.59@tcp2 [ 2571.942096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2572.014165] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2572.019752] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2572.024104] LNet: 120305:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2573.102459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2573.389394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2574.432082] LNet: Removed LNI 192.168.201.59@tcp [ 2574.447586] LNet: Removed LNI 192.168.201.59@tcp2 [ 2574.509469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2574.625186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2574.655167] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2575.613114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2575.635775] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2576.710766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2576.787907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2576.906441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2577.328792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2578.399720] LNet: Removed LNI 192.168.201.59@tcp [ 2579.431904] LNet: Removed LNI 192.168.201.59@tcp2 [ 2579.455146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2579.588436] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2579.605129] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2579.619890] LNet: 120633:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2580.519248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2580.865448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2580.899860] LNetError: 120671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2580.912399] LNetError: 120671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2580.926762] LNet: Removed LNI 192.168.201.59@tcp [ 2582.000505] LNet: Removed LNI 192.168.201.59@tcp2 [ 2582.067784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2582.183621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2582.257695] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2583.239789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2583.255295] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2584.138968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2584.168802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2584.234669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2584.715519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2585.759309] LNet: Removed LNI 192.168.201.59@tcp [ 2585.775495] LNet: Removed LNI 192.168.201.59@tcp2 [ 2585.817781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2585.915175] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2585.920855] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2585.928884] LNet: 120974:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2586.672450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2587.112293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2588.191427] LNet: Removed LNI 192.168.201.59@tcp [ 2588.220350] LNet: Removed LNI 192.168.201.59@tcp2 [ 2588.271067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2588.335080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2588.375065] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2589.559188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2589.574463] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2590.509776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2590.540806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2590.599475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2590.676446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2591.061291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2591.078266] LNet: 121328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2591.085463] LNet: 121328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2592.095241] LNet: Removed LNI 192.168.201.59@tcp [ 2593.128056] LNet: Removed LNI 192.168.201.59@tcp2 [ 2593.154903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2593.286541] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2593.292210] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2593.924614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2602.840086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2604.774777] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:16:57 (1790900217) [ 2605.509498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2605.581415] LNetError: 122104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.590171] LNetError: 122104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2605.608340] LNet: Removed LNI 192.168.201.59@tcp [ 2606.634936] LNet: Removed LNI 192.168.201.59@tcp2 [ 2606.689237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2607.048447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2607.063061] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2607.896415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2607.934802] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2609.067398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2609.094940] LNet: Removed LNI 10.1.2.3@tcp [ 2609.110575] LNet: Removed LNI 192.168.201.59@tcp [ 2609.135328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2609.190360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2609.207231] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2609.214682] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2609.732291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2609.756114] LNet: Removed LNI 192.168.201.59@tcp [ 2609.770771] LNet: Removed LNI 10.1.2.3@tcp [ 2620.084514] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 20:17:12 (1790900232) [ 2620.640697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2620.686718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2620.757883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2620.773485] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2621.535130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2621.572545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2621.634211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2621.659903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2621.688992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2621.713252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2621.741752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2621.768555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2621.799545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2621.830710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2621.861989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2621.908520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2622.162863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2632.024320] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 20:17:24 (1790900244) [ 2632.385508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2632.538033] LNet: 123735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.541940] LNet: 123735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2633.575469] LNet: Removed LNI 192.168.201.59@tcp [ 2633.643705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2633.755674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2633.776100] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2634.500750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2634.530811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2634.568671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.599704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.628701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.655513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.683398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.707857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.759384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.823810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.852330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.877063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.940743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2634.966557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2634.987339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2635.018736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2635.049717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2635.086983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2635.113590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2635.148374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2635.181471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2635.208334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2635.244553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2646.441178] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:17:38 (1790900258) [ 2646.643771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2646.657246] LNetError: 124500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2646.661221] LNetError: 124500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2646.674326] LNet: Removed LNI 192.168.201.59@tcp [ 2646.723137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2646.889239] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2647.040605] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2647.043253] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2647.061824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2649.465462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2649.537405] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2650.834240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2650.866181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2653.791713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2653.900375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2654.021364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.59@tcp -m PUT -e local_timeout -r 1 [ 2654.189532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2654.250973] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2654.256581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2654.782750] LNetError: 124565:0:(lib-move.c:4794:lnet_parse()) 192.168.201.59@tcp, src 192.168.201.59@tcp: Dropping PUT (error -108 looking up sender) [ 2655.779694] LNet: Removed LNI 10.1.2.3@tcp [ 2656.810824] LNet: Removed LNI 192.168.201.59@tcp [ 2657.382779] Key type .llcrypt unregistered [ 2657.392404] Key type ._llcrypt unregistered [ 2673.144713] Key type ._llcrypt registered [ 2673.146857] Key type .llcrypt registered [ 2674.753432] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:18:07 (1790900287) [ 2675.371833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2675.395202] alg: No test for adler32 (adler32-zlib) [ 2676.414649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2676.896821] LNet: 125888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2677.779375] Key type .llcrypt unregistered [ 2677.781465] Key type ._llcrypt unregistered [ 2685.668887] Key type ._llcrypt registered [ 2685.674572] Key type .llcrypt registered [ 2685.999878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2686.021873] alg: No test for adler32 (adler32-zlib) [ 2686.954604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2686.979439] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2695.519875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2695.823401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2696.544763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.159@tcp [ 2696.821340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2696.856332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2696.888620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2696.960745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2697.016985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2697.082269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2697.188599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2697.222019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2697.249800] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 2697.272402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2697.300833] LNet: 126342:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2697.342867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2697.387828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2697.411066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2697.467934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2697.504875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2697.532853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2698.326979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2698.660153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2698.690529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2698.720316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2698.750585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2698.779614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2698.803657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2698.840415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2698.864793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2698.893682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2698.908306] LNet: 126462:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2698.928835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2698.952410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2698.984548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2699.010381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2699.042892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2699.073040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2699.819775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.159@tcp [ 2700.134808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2700.170728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2700.217760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2700.249775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2700.286247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2700.309968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2700.342734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2700.368965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2700.405370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2700.425941] LNet: 126581:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2700.439798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2700.476811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2700.542268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2700.574735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2700.609071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2700.680438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2701.299403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2701.668417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2701.733659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2701.784426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2701.815449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2701.878735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2701.937573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2701.992738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2702.052381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2702.099538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2702.161908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2702.214547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2702.280440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2702.327541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2702.354724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2702.400508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2702.903870] LNet: 126746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2702.912407] LNetError: 126746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2702.939888] LNet: Removed LNI 192.168.201.59@tcp [ 2703.343825] Key type .llcrypt unregistered [ 2703.346162] Key type ._llcrypt unregistered [ 2726.247967] Key type ._llcrypt registered [ 2726.254895] Key type .llcrypt registered [ 2727.642483] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:19:00 (1790900340) [ 2728.191345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2728.213696] alg: No test for adler32 (adler32-zlib) [ 2729.209402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2729.775627] LNet: 127648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2730.275328] Key type .llcrypt unregistered [ 2730.278809] Key type ._llcrypt unregistered [ 2738.965253] Key type ._llcrypt registered [ 2738.967043] Key type .llcrypt registered [ 2739.355243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2739.380912] alg: No test for adler32 (adler32-zlib) [ 2740.352640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2740.388577] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2749.637904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2761.184542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2761.264048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2761.301406] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 2762.457734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2762.824419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2763.944844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.159@tcp [ 2764.328853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2764.367243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2764.414652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2764.477779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2764.535861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2764.584761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2764.642442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2764.704818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2764.774755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2764.808782] LNet: 128311:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2764.847476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2764.930154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2764.959598] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 2765.013963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2765.094700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2765.166258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2765.210755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2766.276439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2766.647543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2766.694224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2766.752105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2766.816929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2766.876931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2766.932422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2766.976396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2767.028371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2767.078243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2767.119714] LNet: 128430:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2767.146353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2767.192446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2767.292337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2767.365335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2767.417725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2767.466394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2768.594280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.159@tcp [ 2768.962354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2768.998239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2769.029865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2769.061615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2769.088432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2769.117383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2769.140338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2769.162902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2769.190487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2769.208157] LNet: 128549:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2769.229320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2769.261811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2769.296156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2769.330807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2769.353727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2769.380543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2770.438437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2771.013426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2771.087242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2771.195929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2771.257613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2771.302213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2771.376276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2771.421203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2771.491722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2771.550589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2771.582286] LNet: 128669:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2771.618421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2771.691909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2771.764810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2771.831214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2771.933674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2772.000852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2778.120357] LNet: 128750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2778.127587] LNetError: 128750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2778.135961] LNet: Removed LNI 192.168.201.59@tcp [ 2778.158926] LNet: Removed LNI 192.168.201.59@tcp1 [ 2778.756912] Key type .llcrypt unregistered [ 2778.759392] Key type ._llcrypt unregistered [ 2803.131668] Key type ._llcrypt registered [ 2803.135333] Key type .llcrypt registered [ 2804.698059] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 20:20:17 (1790900417) [ 2805.565318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2805.594363] alg: No test for adler32 (adler32-zlib) [ 2806.655137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2806.741768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2806.770810] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2807.790735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2807.841937] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2809.291270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2809.462651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2810.527935] LNet: Removed LNI 192.168.201.59@tcp [ 2818.787223] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 20:20:31 (1790900431) [ 2820.994729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2823.279263] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 20:20:36 (1790900436) [ 2832.511904] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 20:20:45 (1790900445) [ 2842.029393] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 20:20:54 (1790900454) [ 2851.291691] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 20:21:04 (1790900464) [ 2861.083852] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 20:21:14 (1790900474) [ 2871.262683] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 20:21:23 (1790900483) [ 2879.681803] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 20:21:32 (1790900492) [ 2887.713855] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:21:40 (1790900500) [ 2897.055864] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:21:50 (1790900510) [ 2905.278330] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:21:58 (1790900518) [ 2913.281930] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:22:06 (1790900526) [ 2922.327900] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:22:15 (1790900535) [ 2932.713431] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:22:25 (1790900545) [ 2940.423160] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:22:33 (1790900553) [ 2948.825739] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:22:41 (1790900561) [ 2957.224416] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:22:49 (1790900569) [ 2966.420853] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:22:59 (1790900579) [ 2975.868337] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:23:08 (1790900588) [ 2976.075899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2976.122564] LNet: 140621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2976.131835] LNetError: 140621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2976.154882] LNet: Removed LNI 192.168.201.59@tcp2 [ 2976.179563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2976.410373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2976.474610] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2978.281316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2978.296948] LNet: 140763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2978.301853] LNetError: 140763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2978.315986] LNet: Removed LNI 192.168.201.59@tcp [ 2978.339104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2978.389579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2978.416742] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2987.246484] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:23:19 (1790900599) [ 2996.562358] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:23:29 (1790900609) [ 3006.051182] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:23:39 (1790900619) [ 3015.222312] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:23:48 (1790900628) [ 3022.193529] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:23:55 (1790900635) [ 3031.663776] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:24:04 (1790900644) [ 3041.754503] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:24:14 (1790900654) [ 3050.601774] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:24:23 (1790900663) [ 3057.825695] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:24:30 (1790900670) [ 3065.992528] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:24:39 (1790900679) [ 3074.353845] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:24:47 (1790900687) [ 3082.237323] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:24:55 (1790900695) [ 3090.715357] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2829 sec ========== 20:25:03 (1790900703) [ 3091.156687] LNet: 148336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3091.160865] LNetError: 148336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3091.172448] LNet: Removed LNI 192.168.201.59@tcp [ 3091.571653] Key type .llcrypt unregistered [ 3091.573114] Key type ._llcrypt unregistered [ 3109.601320] Key type ._llcrypt registered [ 3109.603030] Key type .llcrypt registered [ 3110.301283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3110.320156] alg: No test for adler32 (adler32-zlib) [ 3111.505129] Lustre: Lustre: Build Version: 2.17.59_2_g8e10c0d [ 3111.795634] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 3113.439293] Key type lgssc registered [ 3115.197631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3213.661249] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d [ 3218.461193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3228.131759] Lustre: Unmounted lustre-client [ 3288.071268] Key type lgssc unregistered [ 3288.371091] LNet: 151859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3288.385590] LNetError: 151859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3288.416689] LNet: Removed LNI 192.168.201.59@tcp [ 3289.505514] Key type .llcrypt unregistered [ 3289.509542] Key type ._llcrypt unregistered