[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 621783462 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003138] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008363] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011133] LSM: Security Framework initializing [ 0.013035] Yama: becoming mindful. [ 0.013933] SELinux: Initializing. [ 0.014041] *** VALIDATE selinux *** [ 0.022043] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027327] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030042] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031109] *** VALIDATE tmpfs *** [ 0.032445] *** VALIDATE proc *** [ 0.034128] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.037005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044303] debug: unmapping init [mem 0xffffffff8ac59000-0xffffffff8ac60fff] [ 0.046982] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047697] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048020] ... version: 2 [ 0.049024] ... bit width: 48 [ 0.050009] ... generic registers: 4 [ 0.051009] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055266] rcu: Hierarchical SRCU implementation. [ 0.057324] smp: Bringing up secondary CPUs ... [ 0.058561] x86: Booting SMP configuration: [ 0.059021] .... node #0, CPUs: #1 #2 #3 [ 0.065329] smp: Brought up 1 node, 4 CPUs [ 0.067013] smpboot: Max logical packages: 1 [ 0.068011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.108073] node 0 deferred pages initialised in 38ms [ 0.111096] devtmpfs: initialized [ 0.112230] x86/mm: Memory block size: 128MB [ 0.114995] gcov: version magic: 0x41383552 [ 0.118272] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119063] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.120298] pinctrl core: initialized pinctrl subsystem [ 0.121000] [ 0.123009] ************************************************************* [ 0.125009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127011] ** ** [ 0.130020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.132010] ** ** [ 0.135013] ** This means that this kernel is built to expose internal ** [ 0.137010] ** IOMMU data structures, which may compromise security on ** [ 0.139011] ** your system. ** [ 0.141010] ** ** [ 0.144012] ** If you see this message and you are not debugging the ** [ 0.146009] ** kernel, report this immediately to your vendor! ** [ 0.149012] ** ** [ 0.151017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154013] ************************************************************* [ 0.157471] NET: Registered protocol family 16 [ 0.159429] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.162109] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.165050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168156] cpuidle: using governor menu [ 0.169598] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170413] PCI: Using configuration type 1 for base access [ 0.172201] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187404] cryptd: max_cpu_qlen set to 1000 [ 0.191250] ACPI: Added _OSI(Module Device) [ 0.193010] ACPI: Added _OSI(Processor Device) [ 0.195012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202615] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210075] ACPI: Interpreter enabled [ 0.211051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.213012] ACPI: Using IOAPIC for interrupt routing [ 0.214088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.218574] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229486] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232074] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.238088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.244610] acpiphp: Slot [2] registered [ 0.246101] acpiphp: Slot [5] registered [ 0.247098] acpiphp: Slot [6] registered [ 0.249123] acpiphp: Slot [3] registered [ 0.251094] acpiphp: Slot [4] registered [ 0.252158] acpiphp: Slot [7] registered [ 0.254087] acpiphp: Slot [8] registered [ 0.255172] acpiphp: Slot [9] registered [ 0.257072] acpiphp: Slot [10] registered [ 0.258117] acpiphp: Slot [11] registered [ 0.260097] acpiphp: Slot [12] registered [ 0.261079] acpiphp: Slot [13] registered [ 0.263085] acpiphp: Slot [14] registered [ 0.264093] acpiphp: Slot [15] registered [ 0.266064] acpiphp: Slot [16] registered [ 0.267070] acpiphp: Slot [17] registered [ 0.269099] acpiphp: Slot [18] registered [ 0.271090] acpiphp: Slot [19] registered [ 0.272094] acpiphp: Slot [20] registered [ 0.274105] acpiphp: Slot [21] registered [ 0.276078] acpiphp: Slot [22] registered [ 0.277088] acpiphp: Slot [23] registered [ 0.279093] acpiphp: Slot [24] registered [ 0.280073] acpiphp: Slot [25] registered [ 0.282124] acpiphp: Slot [26] registered [ 0.283090] acpiphp: Slot [27] registered [ 0.285092] acpiphp: Slot [28] registered [ 0.287081] acpiphp: Slot [29] registered [ 0.288106] acpiphp: Slot [30] registered [ 0.290080] acpiphp: Slot [31] registered [ 0.291054] PCI host bridge to bus 0000:00 [ 0.293013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.298115] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.303018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.306043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.308203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.312117] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.315444] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.323014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.328387] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.331017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.334015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.336015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.339603] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.342801] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.347839] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.351896] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.362013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.372644] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.410017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.422017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.455018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.481977] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.496015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.507027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.536055] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.558285] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.560325] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.563350] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.565341] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.568240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.573093] iommu: Default domain type: Passthrough [ 0.575443] SCSI subsystem initialized [ 0.577197] ACPI: bus type USB registered [ 0.579135] usbcore: registered new interface driver usbfs [ 0.581058] usbcore: registered new interface driver hub [ 0.583106] usbcore: registered new device driver usb [ 0.585204] pps_core: LinuxPPS API ver. 1 registered [ 0.587010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.590066] PTP clock support registered [ 0.592117] EDAC MC: Ver: 3.0.0 [ 0.594110] PCI: Using ACPI for IRQ routing [ 0.596716] NetLabel: Initializing [ 0.598008] NetLabel: domain hash size = 128 [ 0.599008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.601071] NetLabel: unlabeled traffic allowed by default [ 0.604166] vgaarb: loaded [ 0.606189] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.609010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.615461] clocksource: Switched to clocksource kvm-clock [ 0.718283] VFS: Disk quotas dquot_6.6.0 [ 0.719507] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.721891] *** VALIDATE ramfs *** [ 0.723122] *** VALIDATE hugetlbfs *** [ 0.724777] pnp: PnP ACPI init [ 0.727092] pnp: PnP ACPI: found 6 devices [ 0.742123] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.745180] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.747150] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.748969] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.751359] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.753644] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.756143] NET: Registered protocol family 2 [ 0.758389] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.763092] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.766845] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.771351] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.775423] TCP: Hash tables configured (established 65536 bind 65536) [ 0.778382] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.781538] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.784450] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.787465] NET: Registered protocol family 1 [ 0.790146] RPC: Registered named UNIX socket transport module. [ 0.792408] RPC: Registered udp transport module. [ 0.794350] RPC: Registered tcp transport module. [ 0.796160] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.798587] NET: Registered protocol family 44 [ 0.800292] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.802549] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.804612] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.806950] PCI: CLS 0 bytes, default 64 [ 0.808691] Unpacking initramfs... [ 2.217209] debug: unmapping init [mem 0xffff94c83cc64000-0xffff94c83ffcffff] [ 2.222731] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.225075] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.228043] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.745160] Initialise system trusted keyrings [ 2.747163] Key type blacklist registered [ 2.749139] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.756993] zbud: loaded [ 2.759795] *** VALIDATE nfs *** [ 2.760965] *** VALIDATE nfs4 *** [ 2.762487] pstore: using deflate compression [ 2.766037] Platform Keyring initialized [ 2.878077] NET: Registered protocol family 38 [ 2.879733] Key type asymmetric registered [ 2.881140] Asymmetric key parser 'x509' registered [ 2.882738] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.885358] io scheduler mq-deadline registered [ 2.886732] io scheduler kyber registered [ 2.888269] io scheduler bfq registered [ 2.890614] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.893746] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.896420] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.899075] ACPI: Power Button [PWRF] [ 2.904534] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.910489] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.934787] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.965581] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.996502] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.002198] Non-volatile memory driver v1.3 [ 3.003740] Linux agpgart interface v0.103 [ 3.034435] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.037065] vda: detected capacity change from 0 to 74711040 [ 3.049742] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.052664] vdb: detected capacity change from 0 to 1073741824 [ 3.059246] libphy: Fixed MDIO Bus: probed [ 3.072687] usbcore: registered new interface driver usbserial_generic [ 3.075311] usbserial: USB Serial support registered for generic [ 3.077704] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.082640] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.084550] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.087093] mousedev: PS/2 mouse device common for all mice [ 3.090364] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.094132] rtc_cmos 00:05: RTC can wake from S4 [ 3.097857] rtc_cmos 00:05: registered as rtc0 [ 3.100320] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.101395] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.101434] intel_pstate: CPU model not supported [ 3.104049] hid: raw HID events driver (C) Jiri Kosina [ 3.114464] usbcore: registered new interface driver usbhid [ 3.117300] usbhid: USB HID core driver [ 3.118757] drop_monitor: Initializing network drop monitor service [ 3.121200] Initializing XFRM netlink socket [ 3.121736] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.124461] NET: Registered protocol family 10 [ 3.127180] Segment Routing with IPv6 [ 3.133112] NET: Registered protocol family 17 [ 3.135583] mpls_gso: MPLS GSO support [ 3.141340] RAS: Correctable Errors collector initialized. [ 3.143393] AVX version of gcm_enc/dec engaged. [ 3.145096] AES CTR mode by8 optimization enabled [ 3.218673] sched_clock: Marking stable (3218651866, 0)->(4080964314, -862312448) [ 3.222233] registered taskstats version 1 [ 3.224080] Loading compiled-in X.509 certificates [ 3.225888] zswap: loaded using pool lzo/zbud [ 3.253437] Key type big_key registered [ 3.266610] Key type encrypted registered [ 3.268239] ima: No TPM chip found, activating TPM-bypass! [ 3.270332] ima: Allocated hash algorithm: sha1 [ 3.272197] ima: No architecture policies found [ 3.274347] evm: Initialising EVM extended attributes: [ 3.276299] evm: security.selinux [ 3.277912] evm: security.ima [ 3.279133] evm: security.capability [ 3.280507] evm: HMAC attrs: 0x1 [ 3.282972] rtc_cmos 00:05: setting system clock to 2026-07-30 07:33:34 UTC (1785396814) [ 3.288979] debug: unmapping init [mem 0xffffffff8bc03000-0xffffffff8bdfffff] [ 3.291914] debug: unmapping init [mem 0xffffffff8a982000-0xffffffff8ac58fff] [ 3.300058] Write protecting the kernel read-only data: 28672k [ 3.303285] debug: unmapping init [mem 0xffffffff89003000-0xffffffff891fffff] [ 3.306040] debug: unmapping init [mem 0xffffffff89914000-0xffffffff899fffff] [ 3.338097] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.344425] systemd[1]: Detected virtualization kvm. [ 3.347030] systemd[1]: Detected architecture x86-64. [ 3.348880] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.371283] systemd[1]: No hostname configured. [ 3.372702] systemd[1]: Set hostname to . [ 3.374584] random: systemd: uninitialized urandom read (16 bytes read) [ 3.376621] systemd[1]: Initializing machine ID from random generator. [ 3.419738] random: ln: uninitialized urandom read (6 bytes read) [ 3.514895] random: systemd: uninitialized urandom read (16 bytes read) [ 3.517226] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.521162] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.525956] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.135378] device-mapper: uevent: version 1.0.3 [ 4.137701] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.879440] random: fast init done [ 4.889819] virtio_net virtio0 ens2: renamed from eth0 [ 4.953307] scsi host0: ata_piix [ 4.988744] scsi host1: ata_piix [ 4.990216] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.992174] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.037322] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.786425] random: crng init done [ 9.787606] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.296146] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.480431] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.762694] SELinux: Disabled at runtime. [ 11.824727] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.834769] systemd[1]: Detected virtualization kvm. [ 11.836902] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.372331] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.376187] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.380542] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.383183] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.387171] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.393552] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.400087] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. Mounting Huge Pages Fil[ 12.555865] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ 12.854148] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.210908] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.277221] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.333669] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.344843] EDAC sbridge: Ver: 1.1.2 [ 14.472384] Key type dns_resolver registered [ 14.782057] NFS: Registering the id_resolver key type [ 14.784087] Key type id_resolver registered [ 14.785343] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg254-client login: [ 41.603404] libcfs: loading out-of-tree module taints kernel. [ 41.712618] Key type ._llcrypt registered [ 41.714996] Key type .llcrypt registered [ 41.971996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.980902] alg: No test for adler32 (adler32-zlib) [ 43.082475] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 43.455864] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 45.104225] Key type lgssc registered [ 45.872361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.600153] hrtimer: interrupt took 14144994 ns [ 237.561537] Lustre: Mounted lustre-client [ 245.057096] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.137179] Lustre: lustre-OST0000-osc-ffff94c88920e000: disconnect after 22s idle [ 264.026656] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing check_logdir /tmp/testlogs/ [ 273.693976] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing yml_node [ 281.546266] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 286.199562] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 290.388195] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 293.579214] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jul 30 03:38:21 EDT 2026 [ 316.292801] Lustre: Unmounted lustre-client [ 349.982213] Key type lgssc unregistered [ 350.191741] LNet: 8384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.212740] LNetError: 8384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 350.248706] LNet: Removed LNI 192.168.202.54@tcp [ 351.024293] Key type .llcrypt unregistered [ 351.026574] Key type ._llcrypt unregistered [ 386.283623] Key type ._llcrypt registered [ 386.286424] Key type .llcrypt registered [ 386.564953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 386.604113] alg: No test for adler32 (adler32-zlib) [ 387.580030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 387.687458] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 387.719894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 393.822429] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 393.936882] LNet: 9262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.954575] LNetError: 9262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 393.983541] LNet: Removed LNI 192.168.202.54@tcp [ 394.561494] Key type .llcrypt unregistered [ 394.566601] Key type ._llcrypt unregistered [ 404.608123] Key type ._llcrypt registered [ 404.612282] Key type .llcrypt registered [ 406.575568] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 03:40:15 (1785397215) [ 407.338459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 407.380823] alg: No test for adler32 (adler32-zlib) [ 408.491539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.709774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.789519] LNet: 9683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.861698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.975324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 409.042613] LNet: 9690:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 420.643613] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 03:40:29 (1785397229) [ 420.779891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.836683] LNet: 10263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.884456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.072868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 432.762937] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 03:40:42 (1785397242) [ 432.970509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.014970] LNet: 10855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.100967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.216623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 441.415698] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 03:40:50 (1785397250) [ 441.618022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.673186] LNet: 11442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.739889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.862368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 452.969762] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 03:41:01 (1785397261) [ 453.287891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.362863] LNet: 12027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.414370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.552608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 454.602772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 465.427882] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 03:41:14 (1785397274) [ 465.638409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.699807] LNet: 12615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.742549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.887418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 466.646189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 467.521022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 478.251085] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 03:41:26 (1785397286) [ 478.392412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.438994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.579266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 488.703250] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 03:41:37 (1785397297) [ 488.856251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.901198] LNet: 13793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 488.918718] LNet: 13793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 488.959355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.118113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 489.192142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 490.091394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 490.142357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 491.482039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 491.604209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 492.573908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 492.640166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 493.751987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 493.810309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 494.420580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 494.482211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 495.324526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 495.411089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 496.363323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 496.420390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 497.726583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 497.818123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 498.783592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 498.828661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 499.469372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 499.524527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 500.270048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 500.338541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 500.998574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 501.026891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 511.862715] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 03:42:00 (1785397320) [ 512.041877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.126823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.279595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 512.398192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 523.573816] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 03:42:12 (1785397332) [ 523.740310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.816313] LNet: 15069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 523.822762] LNet: 15069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 523.884573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.046945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 524.099780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 533.733159] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 03:42:23 (1785397343) [ 533.987194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.054973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.188383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 534.228764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 543.198979] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 03:42:32 (1785397352) [ 543.499859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.574112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.695768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 543.744751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 552.014955] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 03:42:41 (1785397361) [ 552.432833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.501754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.634284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 552.702639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 563.839666] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 03:42:52 (1785397372) [ 564.100678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.284368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.413783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 564.509708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 576.608149] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 03:43:05 (1785397385) [ 576.930710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.988678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.122640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 577.691723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 589.575569] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 03:43:17 (1785397397) [ 589.691898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.717229] LNet: 18600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.724205] LNet: 18600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 589.760264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.062375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 590.621095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 600.354197] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 03:43:29 (1785397409) [ 600.554795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.627064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.870806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 601.057817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 612.239879] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 03:43:41 (1785397421) [ 612.388221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.451717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.579294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 614.137801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 625.343678] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 03:43:54 (1785397434) [ 625.495317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.547208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.679065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 625.814329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 634.862830] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 03:44:04 (1785397444) [ 635.402243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.482589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.657178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 635.782436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 647.994140] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 03:44:17 (1785397457) [ 648.080595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.162402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.323327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 656.492783] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 03:44:26 (1785397466) [ 656.842898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.894855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.995746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 665.907572] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 03:44:35 (1785397475) [ 666.254191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.310747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.468293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 667.575231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 677.771808] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 03:44:46 (1785397486) [ 678.217116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.296412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.422108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 678.489894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 688.800075] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 03:44:57 (1785397497) [ 688.960778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.038369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.175863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 689.233443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 698.483958] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 03:45:08 (1785397508) [ 698.908634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.029368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.202619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 699.299387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 709.852628] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 03:45:19 (1785397519) [ 710.149728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.229216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.286388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 710.322255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 719.799312] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 03:45:29 (1785397529) [ 720.187664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.218854] LNet: 25809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.224537] LNet: 25809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 720.250987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.321796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 720.442984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 720.457234] LNetError: 25821:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 720.492834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 720.550609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 728.010287] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 03:45:37 (1785397537) [ 728.326778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.388711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.544921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MU2PCeeWkn [ 728.688082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MU2PCeeWkn [ 728.787902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MU2PCeeWkn [ 736.046732] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 03:45:45 (1785397545) [ 736.372387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.407714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.470652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 736.510313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 736.584694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 744.801135] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 03:45:53 (1785397553) [ 744.993219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.076362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.223577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 745.267069] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 749.311476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 754.190574] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 756.536172] Key type lgssc registered [ 760.477190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 771.692638] LNetError: 27584:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.54@tcp99 has no available nets [ 778.324996] Key type lgssc unregistered [ 778.867178] LNetError: 28696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.881558] LNet: Removed LNI 192.168.202.54@tcp [ 779.872442] Key type .llcrypt unregistered [ 779.880967] Key type ._llcrypt unregistered [ 790.990504] Key type ._llcrypt registered [ 790.995641] Key type .llcrypt registered [ 793.983299] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 03:46:41 (1785397601) [ 794.596316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 794.650114] alg: No test for adler32 (adler32-zlib) [ 795.719143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.798902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 800.351342] LNet: Message forwarding will be enabled in 2s [ 803.428935] LNet: Message forwarding enabled [ 804.239363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 804.285522] LNet: Message forwarding will stop in 2s to 6s [ 808.066293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 809.589127] LNet: Message forwarding disabled [ 811.102060] LNet: Message forwarding will be enabled in 2s [ 814.754143] LNet: Message forwarding enabled [ 819.981450] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 03:47:09 (1785397629) [ 820.365916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.410144] LNet: 30097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.943990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.153514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 832.565502] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 03:47:21 (1785397641) [ 832.719690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.753305] LNet: 30674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.777383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.876929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 832.974610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 833.046307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 833.114630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 833.189433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 833.352041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 833.402159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 833.432286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 833.465392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 833.485810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 833.508791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 833.530407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 833.585383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 833.639689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 833.679782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 833.707962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 833.798540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 833.916804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 833.979473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 834.027350] LNetError: 30722:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 844.102065] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 03:47:32 (1785397652) [ 844.242404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.284836] LNet: 31289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.347183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.526502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 854.166217] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 03:47:43 (1785397663) [ 857.432433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg254-server [ 860.488937] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 03:47:49 (1785397669) [ 860.622759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.656351] LNet: 32221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.708915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.854102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.882773] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 862.414470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[55-59]@tcp [ 862.485982] LNet: 32361:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 862.636535] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.56@tcp is being used as a gateway but routing feature is not turned on [ 862.665607] LNetError: 32231:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.56@tcp has gone from up to down [ 863.593931] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.56@tcp is being used as a gateway but routing feature is not turned on [ 863.615664] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 864.618575] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.56@tcp is being used as a gateway but routing feature is not turned on [ 864.648275] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 866.676179] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.57@tcp is being used as a gateway but routing feature is not turned on [ 866.696955] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 870.755907] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.56@tcp is being used as a gateway but routing feature is not turned on [ 870.790512] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 11 previous similar messages [ 873.151916] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 03:48:02 (1785397682) [ 876.648380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg254-server [ 878.982984] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.58@tcp is being used as a gateway but routing feature is not turned on [ 878.997185] LNetError: 32231:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 23 previous similar messages [ 880.513507] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 03:48:08 (1785397688) [ 880.719561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.995500] LNetError: 32230:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.202.56@tcp [ 912.800152] LNet: 1 local NIs in recovery (showing 1): 192.168.202.54@tcp [ 914.405790] LNetError: 32236:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.202.59 [ 914.415711] LNetError: Unexpected error -11 connecting to 192.168.202.55@tcp at host 192.168.202.55:988 [ 914.424808] LNetError: 32236:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Skipped 1 previous similar message [ 914.461272] LNetError: Skipped 1 previous similar message [ 914.592148] LNetError: 32231:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.59@tcp has gone from up to down [ 914.604974] LNetError: 32231:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 914.616218] LNet: 33290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.627438] LNetError: 33290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.669134] LNet: Removed LNI 192.168.202.54@tcp [ 914.696015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.774645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.809745] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 916.010068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[55-60/2]@tcp [ 916.051913] LNet: 33422:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 916.072974] LNet: 33422:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 916.106601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[55-60/2]@tcp [ 927.841256] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 03:48:56 (1785397736) [ 928.001518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.648135] LNetError: 33315:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.202.59 [ 967.649547] LNetError: Unexpected error -11 connecting to 192.168.202.55@tcp at host 192.168.202.55:988 [ 967.688883] LNetError: 33315:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Skipped 1 previous similar message [ 967.968728] LNet: 34009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.980462] LNetError: 34009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.022799] LNet: Removed LNI 192.168.202.54@tcp [ 968.088749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.235944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 968.317205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.415138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.522329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 968.617453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.681992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.765191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 968.849446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.900356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.001718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 969.099897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.159893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.265961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 969.405638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.473112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.593620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 969.736537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.786803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.868282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 970.012905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.093468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.208538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 970.373890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.482138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.573051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 970.706748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.774431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.878120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 970.980587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.042367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.161887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 971.288447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.374201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.491742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 980.071435] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 03:49:49 (1785397789) [ 982.900973] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg254-server [ 986.280835] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 03:49:54 (1785397794) [ 988.978940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg254-server [ 992.220447] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 03:50:00 (1785397800) [ 992.483439] LNet: 35417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.493507] LNet: 35417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 993.770377] Key type .llcrypt unregistered [ 993.776305] Key type ._llcrypt unregistered [ 1006.607409] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1006.819942] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1006.823700] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1006.829347] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1009.959799] Key type ._llcrypt registered [ 1009.966493] Key type .llcrypt registered [ 1010.334539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1010.367944] alg: No test for adler32 (adler32-zlib) [ 1011.446410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.534656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1011.576861] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1012.688344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1012.728840] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1016.527753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1016.553715] LNet: Removed LNI 10.1.2.3@tcp [ 1016.775384] LNet: 36147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.800407] LNetError: 36147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.818578] LNet: Removed LNI 192.168.202.54@tcp [ 1018.008536] Key type .llcrypt unregistered [ 1018.013069] Key type ._llcrypt unregistered [ 1035.653164] Key type ._llcrypt registered [ 1035.655437] Key type .llcrypt registered [ 1037.460462] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 03:50:46 (1785397846) [ 1038.192543] Key type .llcrypt unregistered [ 1038.194339] Key type ._llcrypt unregistered [ 1047.341401] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1047.550017] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1047.561655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1047.575541] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1049.473734] Key type ._llcrypt registered [ 1049.482745] Key type .llcrypt registered [ 1049.905400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1049.949606] alg: No test for adler32 (adler32-zlib) [ 1051.132397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.281854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1051.340554] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1052.457492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1052.485691] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1056.923884] LNet: 37637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.953192] LNetError: 37637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1056.980368] LNet: Removed LNI 10.1.2.3@tcp [ 1056.994915] LNet: Removed LNI 192.168.202.54@tcp [ 1057.914330] Key type .llcrypt unregistered [ 1057.915973] Key type ._llcrypt unregistered [ 1077.391462] Key type ._llcrypt registered [ 1077.394573] Key type .llcrypt registered [ 1079.329249] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 03:51:28 (1785397888) [ 1080.132455] Key type .llcrypt unregistered [ 1080.134052] Key type ._llcrypt unregistered [ 1086.676275] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1086.899195] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1086.914902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1086.926506] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1090.028119] Key type ._llcrypt registered [ 1090.030793] Key type .llcrypt registered [ 1090.427616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.468481] alg: No test for adler32 (adler32-zlib) [ 1091.578348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.708709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1091.763868] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1095.280264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1095.320471] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1098.902925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1098.939758] LNet: Removed LNI 10.1.2.3@tcp [ 1098.968849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 1099.060220] LNet: Removed LNI 10.1.2.31@tcp [ 1099.108366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1099.168303] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1100.388681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1100.417778] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1101.414971] LNet: 39559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.426731] LNetError: 39559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 1101.435522] LNet: Removed LNI 10.1.2.31@tcp [ 1101.446809] LNet: Removed LNI 10.1.2.3@tcp [ 1102.112514] Key type .llcrypt unregistered [ 1102.113933] Key type ._llcrypt unregistered [ 1120.577305] Key type ._llcrypt registered [ 1120.578724] Key type .llcrypt registered [ 1122.950558] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 03:52:11 (1785397931) [ 1123.457438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1123.498150] alg: No test for adler32 (adler32-zlib) [ 1124.657363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1124.761916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1124.798903] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1126.077771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1126.132603] LNet: 40515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.144831] LNetError: 40515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1126.172177] LNet: Removed LNI 192.168.202.54@tcp [ 1126.226964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.312951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.54@tcp [ 1126.372472] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1127.909651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.54@tcp [ 1127.971185] LNet: Removed LNI 192.168.202.54@tcp [ 1128.083629] LNet: 40650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.127394] Key type .llcrypt unregistered [ 1129.129871] Key type ._llcrypt unregistered [ 1144.993954] Key type ._llcrypt registered [ 1144.995609] Key type .llcrypt registered [ 1146.958498] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 03:52:35 (1785397955) [ 1150.461769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg254-server [ 1153.709687] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 03:52:42 (1785397962) [ 1154.611857] Key type .llcrypt unregistered [ 1154.617851] Key type ._llcrypt unregistered [ 1168.535228] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1169.020151] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1169.054400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1169.057789] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1172.341320] Key type ._llcrypt registered [ 1172.344539] Key type .llcrypt registered [ 1172.659865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1172.681267] alg: No test for adler32 (adler32-zlib) [ 1173.750715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1173.892982] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 1175.380389] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1179.419063] LNet: 42494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.432075] LNetError: 42494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.440103] LNet: Removed LNI 192.168.202.54@tcp1 [ 1179.456296] LNet: Removed LNI 10.1.2.3@tcp2 [ 1180.574362] Key type .llcrypt unregistered [ 1180.578433] Key type ._llcrypt unregistered [ 1199.649857] Key type ._llcrypt registered [ 1199.658109] Key type .llcrypt registered [ 1201.589561] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 03:53:30 (1785398010) [ 1202.380591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1202.434357] alg: No test for adler32 (adler32-zlib) [ 1203.547664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1203.639970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1203.686642] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1205.386904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1205.455021] LNet: 43468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.478378] LNetError: 43468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1205.528886] LNet: Removed LNI 192.168.202.54@tcp [ 1205.577901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1205.748209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1205.873621] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1216.338394] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 03:53:45 (1785398025) [ 1216.584724] LNet: 44059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.609833] LNetError: 44059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.639713] LNet: Removed LNI 192.168.202.54@tcp [ 1217.912680] Key type .llcrypt unregistered [ 1217.917163] Key type ._llcrypt unregistered [ 1227.993609] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1228.211046] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1228.225420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1228.236144] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1231.442420] Key type ._llcrypt registered [ 1231.446074] Key type .llcrypt registered [ 1231.836648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1231.881938] alg: No test for adler32 (adler32-zlib) [ 1233.000231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1233.072035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1233.121722] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1234.374541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1234.397347] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1237.640404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1237.679143] LNet: 44806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.692137] LNetError: 44806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1237.694796] LNet: Removed LNI 10.1.2.3@tcp [ 1237.710110] LNet: Removed LNI 192.168.202.54@tcp [ 1237.741283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.831594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1237.863098] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1237.879151] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1238.830294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1238.887104] LNet: 44833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.898960] LNetError: 44833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1238.905581] LNetError: 44833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1238.920568] LNet: Removed LNI 10.1.2.3@tcp [ 1238.940774] LNet: Removed LNI 192.168.202.54@tcp [ 1239.004312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1239.107200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1239.135919] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1239.145748] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1240.129980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1240.199651] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 1242.253670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1242.310664] LNet: 44996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.326272] LNetError: 44996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1242.335727] LNetError: 44996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1242.353355] LNet: Removed LNI 10.1.2.3@tcp [ 1242.355055] LNet: Removed LNI 192.168.202.54@tcp [ 1242.393239] LNet: Removed LNI 192.168.202.54@tcp2 [ 1242.446709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.609862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1242.670947] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1242.678846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1242.707888] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 1253.475376] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 03:54:23 (1785398063) [ 1254.037962] LNet: 45595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1254.052444] LNetError: 45595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1254.068225] LNetError: 45595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1254.100614] LNet: Removed LNI 10.1.2.3@tcp [ 1254.101968] LNet: Removed LNI 192.168.202.54@tcp [ 1254.117286] LNet: Removed LNI 192.168.202.54@tcp2 [ 1255.019072] Key type .llcrypt unregistered [ 1255.027036] Key type ._llcrypt unregistered [ 1264.537364] Key type ._llcrypt registered [ 1264.543793] Key type .llcrypt registered [ 1264.805465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1264.847089] alg: No test for adler32 (adler32-zlib) [ 1265.829797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1265.913116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1265.957786] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1267.117721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1267.176374] LNet: 45990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.186671] LNetError: 45990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.221304] LNet: Removed LNI 192.168.202.54@tcp [ 1267.258553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1267.400164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1267.452832] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1268.172686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1268.209715] LNet: 46020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.233431] LNetError: 46020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.258399] LNet: Removed LNI 192.168.202.54@tcp [ 1268.319940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1268.413927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1268.477807] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1269.161506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1269.195699] LNet: Removed LNI 192.168.202.54@tcp [ 1269.232012] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.276977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1269.295410] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1269.836124] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1270.108923] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1270.118086] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1270.126678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1272.522590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1272.552542] LNet: 46129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.560269] LNet: 46129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1272.568253] LNetError: 46129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.572877] LNetError: 46129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1272.586653] LNet: Removed LNI 192.168.202.54@tcp [ 1272.604286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.693482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1272.709871] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1281.961984] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 03:54:51 (1785398091) [ 1282.304261] LNet: 46726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.312665] LNetError: 46726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.349608] LNet: Removed LNI 192.168.202.54@tcp [ 1283.214820] Key type .llcrypt unregistered [ 1283.218978] Key type ._llcrypt unregistered [ 1292.245339] Key type ._llcrypt registered [ 1292.248669] Key type .llcrypt registered [ 1292.611532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1292.648468] alg: No test for adler32 (adler32-zlib) [ 1293.607785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1293.670826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1293.711653] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1294.782472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1294.831713] LNet: 47126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.841958] LNetError: 47126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.865184] LNet: Removed LNI 192.168.202.54@tcp [ 1294.910910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1295.009289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1295.075171] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1304.689152] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 03:55:13 (1785398113) [ 1304.935523] LNet: 47716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.944643] LNetError: 47716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.982986] LNet: Removed LNI 192.168.202.54@tcp [ 1306.230421] Key type .llcrypt unregistered [ 1306.234963] Key type ._llcrypt unregistered [ 1320.100494] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1320.194340] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1320.204516] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1320.217441] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1323.096046] Key type ._llcrypt registered [ 1323.098697] Key type .llcrypt registered [ 1323.519685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1323.536790] alg: No test for adler32 (adler32-zlib) [ 1324.501378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.562129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1324.583462] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1325.702156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1325.719354] LNet: 48171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.728995] LNetError: 48171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.746844] LNet: Removed LNI 192.168.202.54@tcp [ 1325.759837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.849465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1325.881373] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1334.828569] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 03:55:43 (1785398143) [ 1335.221724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1335.298427] LNet: 48768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.314469] LNetError: 48768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.384461] LNet: Removed LNI 192.168.202.54@tcp [ 1335.470273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.785740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1345.953107] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 03:55:54 (1785398154) [ 1346.128189] LNet: 49345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.259180] Key type .llcrypt unregistered [ 1347.261868] Key type ._llcrypt unregistered [ 1358.550718] Key type ._llcrypt registered [ 1358.555201] Key type .llcrypt registered [ 1358.857721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1358.895862] alg: No test for adler32 (adler32-zlib) [ 1359.947185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.044410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1360.095677] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1361.607752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1361.639555] LNet: 49752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.649409] LNetError: 49752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1361.674527] LNet: Removed LNI 192.168.202.54@tcp [ 1361.700081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1361.795498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1361.831895] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1371.412030] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 03:56:20 (1785398180) [ 1371.515382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1371.560916] LNet: 50338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.575340] LNetError: 50338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.633253] LNet: Removed LNI 192.168.202.54@tcp [ 1371.667139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.800094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1380.042616] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 03:56:29 (1785398189) [ 1382.546868] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1384.238190] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 03:56:34 (1785398194) [ 1385.865667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1388.589995] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 03:56:37 (1785398197) [ 1390.919553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1392.687766] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 03:56:42 (1785398202) [ 1394.673512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1397.013235] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 03:56:46 (1785398206) [ 1399.788927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1402.235720] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 03:56:51 (1785398211) [ 1402.499280] LNet: 52393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.424454] Key type .llcrypt unregistered [ 1403.427500] Key type ._llcrypt unregistered [ 1413.428582] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1413.719496] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1413.723884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1413.742924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1416.853463] Key type ._llcrypt registered [ 1416.859358] Key type .llcrypt registered [ 1417.286952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1417.332090] alg: No test for adler32 (adler32-zlib) [ 1418.440868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.616989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1418.683202] LNet: Added LNI 192.168.202.54@tcp [8/512/0/180] [ 1418.703803] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1418.705916] LNet: 52714:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1418.894505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1418.951789] LNet: 52733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1418.967377] LNetError: 52733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1418.983414] LNet: Removed LNI 10.1.2.3@tcp [ 1418.992100] LNetError: 52733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.027210] LNet: Removed LNI 192.168.202.54@tcp [ 1419.100538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.217671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1419.266051] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1419.348614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1419.400403] LNet: 52757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.409158] LNetError: 52757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.449378] LNet: Removed LNI 192.168.202.54@tcp [ 1419.494093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.635436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1419.672176] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1419.679817] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1419.685538] LNet: 52765:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1419.936904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.003349] LNet: 52783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.027404] LNetError: 52783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1420.030630] LNet: Removed LNI 10.1.2.3@tcp [ 1420.046594] LNetError: 52783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.079263] LNet: Removed LNI 192.168.202.54@tcp [ 1420.111988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.238667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1420.293751] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1420.309566] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1420.314593] LNet: 52791:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1420.656402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.721181] LNet: 52817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.733923] LNetError: 52817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1420.749117] LNet: Removed LNI 10.1.2.3@tcp [ 1420.750609] LNetError: 52817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.793601] LNet: Removed LNI 192.168.202.54@tcp [ 1420.823335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.948911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1420.998863] LNet: Added LNI 192.168.202.54@tcp [8/256/0/182] [ 1421.006607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1421.190690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1421.243795] LNet: 52846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.266413] LNetError: 52846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1421.280986] LNet: Removed LNI 10.1.2.3@tcp [ 1421.285707] LNetError: 52846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.330406] LNet: Removed LNI 192.168.202.54@tcp [ 1421.403691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.541705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1421.581895] LNet: Added LNI 192.168.202.54@tcp [8/256/0/182] [ 1421.588772] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1421.592886] LNet: 52854:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1421.777595] LNet: Removed LNI 10.1.2.3@tcp [ 1421.797449] LNet: Removed LNI 192.168.202.54@tcp [ 1422.937669] Key type .llcrypt unregistered [ 1422.944579] Key type ._llcrypt unregistered [ 1445.779736] Key type ._llcrypt registered [ 1445.785870] Key type .llcrypt registered [ 1448.460929] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 03:57:36 (1785398256) [ 1449.180463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1449.203570] alg: No test for adler32 (adler32-zlib) [ 1450.302197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.455149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.540560] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1451.921466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.54 [ 1452.012301] LNet: 53832:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1452.039405] LNet: Removed LNI 192.168.202.54@tcp [ 1462.101941] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 03:57:51 (1785398271) [ 1462.522489] LNet: 54402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.551934] LNetError: 54402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1462.610625] LNet: Removed LNI 192.168.202.54@tcp [ 1463.433678] Key type .llcrypt unregistered [ 1463.437286] Key type ._llcrypt unregistered [ 1472.027730] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1472.312621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1475.645105] Key type ._llcrypt registered [ 1475.654673] Key type .llcrypt registered [ 1476.109986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1476.152670] alg: No test for adler32 (adler32-zlib) [ 1477.953848] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1487.142782] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 03:58:16 (1785398296) [ 1487.242314] LNet: 55315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.247693] LNetError: 55315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1487.296338] LNet: Removed LNI 10.1.2.3@tcp [ 1488.204437] Key type .llcrypt unregistered [ 1488.208345] Key type ._llcrypt unregistered [ 1496.595289] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1496.964249] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1496.973329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1496.979346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1500.106172] Key type ._llcrypt registered [ 1500.110925] Key type .llcrypt registered [ 1500.449790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1500.506418] alg: No test for adler32 (adler32-zlib) [ 1501.697730] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1509.873722] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 03:58:39 (1785398319) [ 1512.014971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1514.053482] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 03:58:43 (1785398323) [ 1514.453817] LNet: 56509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.464266] LNetError: 56509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1514.483440] LNet: Removed LNI 10.1.2.3@tcp [ 1515.342586] Key type .llcrypt unregistered [ 1515.346102] Key type ._llcrypt unregistered [ 1525.373355] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1525.710294] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1525.718976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1525.727729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1528.541656] Key type ._llcrypt registered [ 1528.545797] Key type .llcrypt registered [ 1528.770894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1528.784915] alg: No test for adler32 (adler32-zlib) [ 1529.812887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.898461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1542.773893] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 03:59:11 (1785398351) [ 1543.221919] LNet: 57739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.229709] LNetError: 57739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1543.250149] LNet: Removed LNI 10.1.2.3@tcp [ 1544.037371] Key type .llcrypt unregistered [ 1544.039695] Key type ._llcrypt unregistered [ 1553.214501] Key type ._llcrypt registered [ 1553.226533] Key type .llcrypt registered [ 1553.524121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1553.555927] alg: No test for adler32 (adler32-zlib) [ 1554.548479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1554.602058] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1564.733440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 1566.120202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_interrupt -r 1 [ 1566.215443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_interrupt -r 1 [ 1566.310094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1566.350121] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1566.761588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1566.816256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1566.897503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1566.945070] LNet: 1 local NIs in recovery (showing 1): 192.168.202.54@tcp [ 1566.983539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1567.045763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1567.123472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1567.200416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1567.277814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1567.337347] LNet: 58179:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1567.375428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1567.443937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1567.530602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1567.618607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1567.689464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1567.758288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1567.844426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1568.964254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_dropped -r 1 [ 1569.049176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_dropped -r 1 [ 1569.142693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1569.209436] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1569.613480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1569.706694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1569.767812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1569.813824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1569.888246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1569.983969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1570.065611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1570.134329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1570.171563] LNet: 58289:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1570.212113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1570.267668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1570.351493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1570.411427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1570.482527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1570.561954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1570.624427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1571.484839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_aborted -r 1 [ 1571.572457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_aborted -r 1 [ 1571.664775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1571.706678] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1571.714635] LNet: Skipped 1 previous similar message [ 1572.080435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1572.122249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1572.170869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1572.225409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1572.287583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1572.374273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1572.437634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1572.521917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1572.576174] LNet: 58398:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1572.618942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1572.718481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1572.803972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1572.851561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1572.912263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1572.990223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1573.063628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1574.286597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_no_route -r 1 [ 1574.366593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_no_route -r 1 [ 1574.449905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1574.514694] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1574.964915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1575.019193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1575.080355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1575.173533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1575.254184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1575.369225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1575.495475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1575.590438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1575.653601] LNet: 58508:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1575.718528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1575.809609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1575.907067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1575.995895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1576.083532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1576.174672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1576.263919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1577.459273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_timeout -r 1 [ 1577.572140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_timeout -r 1 [ 1577.685070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1578.297037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1578.345310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1578.413537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1578.478419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1578.609527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1578.714627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1578.799804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1578.858668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1578.927035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1578.982169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1579.068756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1579.130819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1579.211689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1579.276421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1579.371049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1580.423899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_error -r 1 [ 1580.474393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_error -r 1 [ 1580.568911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1580.593418] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1580.596022] LNet: Skipped 2 previous similar messages [ 1580.990187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1581.041610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1581.105563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1581.162160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1581.195374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1581.238557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1581.325966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1581.382496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1581.410232] LNet: 58727:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1581.417995] LNet: 58727:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1581.457757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1581.531878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1581.570542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1581.603889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1581.663401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1581.732289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1581.776430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1582.220923] LNet: 58768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.238285] LNetError: 58768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.261837] LNet: Removed LNI 192.168.202.54@tcp [ 1582.968587] Key type .llcrypt unregistered [ 1582.971428] Key type ._llcrypt unregistered [ 1608.178227] Key type ._llcrypt registered [ 1608.179372] Key type .llcrypt registered [ 1610.108205] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:00:19 (1785398419) [ 1611.680560] Key type .llcrypt unregistered [ 1611.685090] Key type ._llcrypt unregistered [ 1620.914401] Key type ._llcrypt registered [ 1620.916517] Key type .llcrypt registered [ 1621.151449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1621.162599] alg: No test for adler32 (adler32-zlib) [ 1622.127486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1622.173297] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1634.159697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 1648.001921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.104792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1648.171485] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 1651.112020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_interrupt -r 1 [ 1651.176587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_interrupt -r 1 [ 1651.241861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_interrupt -r 1 [ 1651.308278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_interrupt -r 1 [ 1651.391616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_interrupt -r 1 [ 1651.472253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_interrupt -r 1 [ 1651.537518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_interrupt -r 1 [ 1651.625820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_interrupt -r 1 [ 1651.695654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1651.742438] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1651.752568] LNet: 1 local NIs in recovery (showing 1): 192.168.202.54@tcp [ 1652.108168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1652.182934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1652.249343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1652.292791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1652.388464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1652.484483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1652.578694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1652.646524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1652.686154] LNet: 60253:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1652.737613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1652.778254] LNet: There was an unexpected network error while writing to 192.168.202.54: rc = -22 [ 1652.789463] LNet: Skipped 3 previous similar messages [ 1652.796631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1652.858476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1652.908601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1652.976849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1653.053661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1653.114849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1653.898452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_dropped -r 1 [ 1653.964732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_dropped -r 1 [ 1654.047807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_dropped -r 1 [ 1654.126630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_dropped -r 1 [ 1654.197438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_dropped -r 1 [ 1654.290740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_dropped -r 1 [ 1654.359568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_dropped -r 1 [ 1654.430610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_dropped -r 1 [ 1654.515513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1654.556969] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1654.982382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1655.021063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1655.097530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1655.153157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1655.205334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1655.244712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1655.299085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1655.350669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1655.379855] LNet: 60375:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1655.449688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1655.515493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1655.564599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1655.608457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1655.655376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1655.700657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1655.756495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1656.723379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_aborted -r 1 [ 1656.783985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_aborted -r 1 [ 1656.874179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_aborted -r 1 [ 1656.969908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_aborted -r 1 [ 1657.047241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_aborted -r 1 [ 1657.155492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_aborted -r 1 [ 1657.204545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_aborted -r 1 [ 1657.293460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_aborted -r 1 [ 1657.369588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1657.399250] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1657.409124] LNet: Skipped 2 previous similar messages [ 1657.816221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1657.879457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1657.958132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1658.026761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1658.078564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1658.162870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1658.234347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1658.288367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1658.321472] LNet: 60496:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1658.349781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1658.398386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1658.438647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1658.489194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1658.541790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1658.587302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1658.623548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1659.412909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_no_route -r 1 [ 1659.442616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_no_route -r 1 [ 1659.481500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_no_route -r 1 [ 1659.531123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_no_route -r 1 [ 1659.581462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_no_route -r 1 [ 1659.637662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_no_route -r 1 [ 1659.685694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_no_route -r 1 [ 1659.725496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_no_route -r 1 [ 1659.778979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1660.098940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1660.153406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1660.194498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1660.235814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1660.293683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1660.369782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1660.470381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1660.535317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1660.591486] LNet: 60617:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1660.638800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1660.713390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1660.766747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1660.821673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1660.910655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1660.980434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1661.038255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1661.725485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_timeout -r 1 [ 1661.783757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_timeout -r 1 [ 1661.869443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_timeout -r 1 [ 1661.964623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_timeout -r 1 [ 1662.059127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_timeout -r 1 [ 1662.159406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_timeout -r 1 [ 1662.236888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_timeout -r 1 [ 1662.347335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_timeout -r 1 [ 1662.459996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1662.533194] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1662.548887] LNet: Skipped 5 previous similar messages [ 1662.888365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1662.928912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1662.956781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1663.006784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1663.048946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1663.103177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1663.162567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1663.241183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1663.314229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1663.386240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1663.458207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1663.508496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1663.541978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1663.602500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1663.682619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1664.533812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_error -r 1 [ 1664.625251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_error -r 1 [ 1664.706772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_error -r 1 [ 1664.771414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_error -r 1 [ 1664.834360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_error -r 1 [ 1664.890395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_error -r 1 [ 1664.958553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_error -r 1 [ 1665.027295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_error -r 1 [ 1665.094916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1665.483883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1665.529782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1665.568836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1665.613496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1665.654202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1665.713246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1665.785585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1665.862337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1665.910491] LNet: 60860:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1665.924370] LNet: 60860:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1665.964386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1666.042178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1666.118901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1666.175259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1666.314867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1666.400058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1666.500659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1675.207529] LNet: 60936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.231721] LNetError: 60936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.249818] LNet: Removed LNI 192.168.202.54@tcp [ 1676.337086] LNet: Removed LNI 192.168.202.54@tcp1 [ 1677.238385] Key type .llcrypt unregistered [ 1677.243132] Key type ._llcrypt unregistered [ 1704.108462] Key type ._llcrypt registered [ 1704.110957] Key type .llcrypt registered [ 1705.861780] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:01:55 (1785398515) [ 1706.929644] Key type .llcrypt unregistered [ 1706.938097] Key type ._llcrypt unregistered [ 1716.221766] Key type ._llcrypt registered [ 1716.227722] Key type .llcrypt registered [ 1716.655707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1716.705072] alg: No test for adler32 (adler32-zlib) [ 1717.768596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1717.813651] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1728.984373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 1729.679896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e remote_dropped -r 1 [ 1729.714272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e remote_dropped -r 1 [ 1729.742960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1729.764243] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1729.987634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1730.015375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1730.052323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1730.088683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1730.116938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1730.148928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1730.188458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1730.220566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1730.232559] LNet: 62188:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1730.246803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1730.268882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1730.303580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1730.326903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1730.350724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1730.372600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1730.426151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1730.953833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e remote_error -r 1 [ 1730.985770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e remote_error -r 1 [ 1731.020828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1731.049793] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1731.293094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1731.332536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1731.381759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1731.429740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1731.470520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1731.516807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1731.555651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1731.602321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1731.628258] LNet: 62297:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1731.650411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1731.708533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1731.748493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1731.805626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1731.856555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1731.895363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1731.938820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1732.657736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e remote_timeout -r 1 [ 1732.708666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e remote_timeout -r 1 [ 1732.769643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1732.812263] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1733.096138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1733.138559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1733.189637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1733.249857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1733.325755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1733.394800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1733.450797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1733.513101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1733.547213] LNet: 62406:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1733.580518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1733.617670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1733.669829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1733.722748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1733.767737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1733.830739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1733.899911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1734.297944] LNet: 62448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.310871] LNetError: 62448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.338274] LNet: Removed LNI 192.168.202.54@tcp [ 1734.890744] Key type .llcrypt unregistered [ 1734.895433] Key type ._llcrypt unregistered [ 1756.852347] Key type ._llcrypt registered [ 1756.853514] Key type .llcrypt registered [ 1758.600645] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:02:47 (1785398567) [ 1760.011344] Key type .llcrypt unregistered [ 1760.013106] Key type ._llcrypt unregistered [ 1771.262498] Key type ._llcrypt registered [ 1771.271494] Key type .llcrypt registered [ 1771.717381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1771.762131] alg: No test for adler32 (adler32-zlib) [ 1772.864755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1772.942741] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1785.570892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 1805.288519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.356447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1805.397648] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 1806.899992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e remote_dropped -r 1 [ 1806.946501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e remote_dropped -r 1 [ 1807.068390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e remote_dropped -r 1 [ 1807.164062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e remote_dropped -r 1 [ 1807.230564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e remote_dropped -r 1 [ 1807.352292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e remote_dropped -r 1 [ 1807.434877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e remote_dropped -r 1 [ 1807.508204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e remote_dropped -r 1 [ 1807.618658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1807.677176] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1807.680175] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.154@tcp [ 1808.030673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1808.108908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1808.161178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1808.231449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1808.270554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1808.329202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1808.380662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1808.443151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1808.468537] LNet: 63917:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1808.510593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1808.551457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1808.600271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1808.648883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1808.704897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1808.737432] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1808.742091] LNet: Skipped 3 previous similar messages [ 1808.768307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1808.804583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1808.876485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1809.494484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e remote_error -r 1 [ 1809.588163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e remote_error -r 1 [ 1809.669179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e remote_error -r 1 [ 1809.738220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e remote_error -r 1 [ 1809.814592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e remote_error -r 1 [ 1809.868511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e remote_error -r 1 [ 1809.920767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e remote_error -r 1 [ 1809.958201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e remote_error -r 1 [ 1810.008366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1810.031527] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1810.386025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1810.440373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1810.485377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1810.530817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1810.563797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1810.636610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1810.689675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1810.741637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1810.767561] LNet: 64040:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1810.803996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1810.840772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1810.901199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1810.948055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1810.995450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1811.053948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1811.116198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1811.204554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1811.987811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e remote_timeout -r 1 [ 1812.061597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e remote_timeout -r 1 [ 1812.119283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e remote_timeout -r 1 [ 1812.167084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e remote_timeout -r 1 [ 1812.212892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e remote_timeout -r 1 [ 1812.294368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e remote_timeout -r 1 [ 1812.362389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e remote_timeout -r 1 [ 1812.418842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e remote_timeout -r 1 [ 1812.506101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.154@tcp [ 1812.558450] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1812.888103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1812.934472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1812.967530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1813.023842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1813.078319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1813.130448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1813.185316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1813.235316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1813.267831] LNet: 64164:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1813.291891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1813.331188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1813.374932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1813.417939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1813.457183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1813.500237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1813.552689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1813.598989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1819.582760] LNet: 64242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.594080] LNetError: 64242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.607651] LNet: Removed LNI 192.168.202.54@tcp [ 1819.631282] LNet: Removed LNI 192.168.202.54@tcp1 [ 1820.536809] Key type .llcrypt unregistered [ 1820.542669] Key type ._llcrypt unregistered [ 1850.633341] Key type ._llcrypt registered [ 1850.635316] Key type .llcrypt registered [ 1851.916474] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:04:21 (1785398661) [ 1854.237527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1857.176066] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:04:26 (1785398666) [ 1858.870676] Key type .llcrypt unregistered [ 1858.875730] Key type ._llcrypt unregistered [ 1870.160355] Key type ._llcrypt registered [ 1870.162387] Key type .llcrypt registered [ 1870.447851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1870.486347] alg: No test for adler32 (adler32-zlib) [ 1871.507860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1871.561113] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1880.580293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 1881.664266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e network_timeout -r 1 [ 1881.748327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e network_timeout -r 1 [ 1881.830395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 1881.858231] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1882.080650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1882.104150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1882.124935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1882.148549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1882.169697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1882.188450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1882.210922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1882.234480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1882.247699] LNet: 65790:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1882.264174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1882.308525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1882.355750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1882.405478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1882.449614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1882.515755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1882.812557] LNet: 65830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.828491] LNetError: 65830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.869964] LNet: Removed LNI 192.168.202.54@tcp [ 1883.616375] Key type .llcrypt unregistered [ 1883.622248] Key type ._llcrypt unregistered [ 1914.061542] Key type ._llcrypt registered [ 1914.063350] Key type .llcrypt registered [ 1914.526240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1914.578558] alg: No test for adler32 (adler32-zlib) [ 1915.758452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1915.833055] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1926.493130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 1943.172855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.247859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1943.285449] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 1945.268444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e network_timeout -r 1 [ 1945.335360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e network_timeout -r 1 [ 1945.411301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e network_timeout -r 1 [ 1945.506900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e network_timeout -r 1 [ 1945.609597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e network_timeout -r 1 [ 1945.686294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e network_timeout -r 1 [ 1945.775473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e network_timeout -r 1 [ 1945.867188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e network_timeout -r 1 [ 1946.006948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 1946.066730] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1946.592787] LNet: 1 local NIs in recovery (showing 1): 192.168.202.54@tcp [ 1946.607652] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.154@tcp [ 1946.667249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1946.777231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1946.843788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1946.932105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1947.025494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1947.093087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1947.207690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1947.285542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1947.341114] LNet: 66712:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1947.393295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1947.460069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1947.524592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1947.635435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1947.680865] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1947.697341] LNet: Skipped 1 previous similar message [ 1947.734878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1947.816664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1954.259113] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 1954.924664] LNet: 66786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.933768] LNetError: 66786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.000404] LNet: Removed LNI 192.168.202.54@tcp [ 1956.001939] LNetError: 66786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.026334] LNet: Removed LNI 192.168.202.54@tcp1 [ 1956.833526] Key type .llcrypt unregistered [ 1956.835977] Key type ._llcrypt unregistered [ 1989.855891] Key type ._llcrypt registered [ 1989.863672] Key type .llcrypt registered [ 1991.994596] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:06:40 (1785398800) [ 1992.430957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1992.449688] alg: No test for adler32 (adler32-zlib) [ 1993.433600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.539217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1993.587774] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 1994.625862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1994.673558] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 1995.570379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 1995.645153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1995.695752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1995.739675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1995.785114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1995.821848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1995.863117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.54@tcp [ 1995.901241] LNetError: 67889:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.54@tcp to recovery [ 1996.576260] LNet: 1 local NIs in recovery (showing 1): 192.168.202.54@tcp [ 1996.576506] LNet: There was an unexpected network error while writing to 192.168.202.54: rc = -22 [ 1998.624402] LNet: There was an unexpected network error while writing to 192.168.202.54: rc = -22 [ 1999.561989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2002.720493] LNet: There was an unexpected network error while writing to 192.168.202.54: rc = -22 [ 2003.264793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2003.302835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2003.391973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2003.428058] LNet: 67996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.438561] LNetError: 67996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.449660] LNet: Removed LNI 192.168.202.54@tcp [ 2003.486588] LNet: Removed LNI 192.168.202.54@tcp1 [ 2003.531863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.635979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2003.688175] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2004.817864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2004.863470] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 2006.153303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 2006.249102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2006.372434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 2006.435180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 2006.488834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 2006.544298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2006.590087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2006.653075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.54@tcp [ 2006.689845] LNetError: 68269:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.54@tcp to recovery [ 2007.712531] LNet: There was an unexpected network error while writing to 192.168.202.54: rc = -22 [ 2010.504465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2013.871253] LNet: There was an unexpected network error while writing to 192.168.202.54: rc = -22 [ 2013.875047] LNet: Skipped 1 previous similar message [ 2019.005965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2019.037329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2019.083453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2027.369996] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:07:17 (1785398837) [ 2027.708861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2027.775339] LNet: 68988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.787712] LNetError: 68988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.800158] LNetError: 68988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2027.815314] LNet: Removed LNI 192.168.202.54@tcp [ 2027.837743] LNet: Removed LNI 192.168.202.54@tcp1 [ 2027.878701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.960418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2028.004434] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2028.913388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2028.949927] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 2029.910295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 2030.003636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 2030.046975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 2030.094291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 2030.118832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2030.163978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2030.210938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.54@tcp --health 0 [ 2030.321912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2031.013445] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.54@tcp [ 2031.013760] LNet: There was an unexpected network error while writing to 192.168.202.54: rc = -22 [ 2031.031717] LNet: Skipped 1 previous similar message [ 2046.600815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2046.802530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2046.841142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2046.873441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.54@tcp --health 500 [ 2047.061685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2052.863709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2052.992798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.009103] LNet: 69472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.016982] LNetError: 69472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.022158] LNetError: 69472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2053.028531] LNet: Removed LNI 192.168.202.54@tcp [ 2053.044641] LNet: Removed LNI 192.168.202.54@tcp1 [ 2053.084918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.172882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2053.219812] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2054.027112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2054.071119] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 2055.419410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 2055.482183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2055.594685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 2055.660841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 2055.714908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 2055.782136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2055.851280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2055.912131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.54@tcp --health 0 [ 2056.225645] LNet: There was an unexpected network error while writing to 192.168.202.54: rc = -22 [ 2056.238147] LNet: Skipped 3 previous similar messages [ 2057.432175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2067.412830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2067.474198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2067.528820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2077.670579] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:08:06 (1785398886) [ 2078.307154] LNet: 70509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2078.327235] LNetError: 70509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2078.337888] LNetError: 70509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2078.355612] LNet: Removed LNI 192.168.202.54@tcp [ 2078.372131] LNet: Removed LNI 192.168.202.54@tcp1 [ 2079.152542] Key type .llcrypt unregistered [ 2079.154447] Key type ._llcrypt unregistered [ 2088.399373] Key type ._llcrypt registered [ 2088.400775] Key type .llcrypt registered [ 2088.795131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2088.832262] alg: No test for adler32 (adler32-zlib) [ 2089.882296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2089.950273] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2103.927432] Lustre: DEBUG MARKER: Initial discovery [ 2104.050114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.154@tcp [ 2107.545168] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 2107.707762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 2107.772618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.154@tcp [ 2107.801466] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 2107.833368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2107.877692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.154@tcp -r 1 -m GET -l 3 [ 2107.918888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.154@tcp -r 1 -m GET -e local_error [ 2107.971154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.154@tcp -r 1 -m PUT -l 6 [ 2108.388254] LNet: 1 local NIs in recovery (showing 1): 192.168.202.54@tcp [ 2109.763206] Lustre: DEBUG MARKER: Force oleg254-client.virtnet to discover 192.168.202.154@tcp (in background) [ 2110.234758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.154@tcp [ 2111.873122] Lustre: DEBUG MARKER: Force oleg254-server to discover 192.168.202.54@tcp [ 2113.312862] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 2115.583390] Lustre: DEBUG MARKER: Wait for 71274 [ 2116.900127] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 2117.525492] Lustre: DEBUG MARKER: Finished wait on 71274 [ 2117.825349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 2117.877551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2118.240133] LNet: 71722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.244625] LNetError: 71722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.269392] LNet: Removed LNI 192.168.202.54@tcp [ 2118.767395] Key type .llcrypt unregistered [ 2118.769339] Key type ._llcrypt unregistered [ 2142.166174] Key type ._llcrypt registered [ 2142.168171] Key type .llcrypt registered [ 2143.643356] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:09:13 (1785398953) [ 2144.548324] Key type .llcrypt unregistered [ 2144.551252] Key type ._llcrypt unregistered [ 2154.263332] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2154.551175] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2154.566477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2154.577699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2157.701322] Key type ._llcrypt registered [ 2157.707233] Key type .llcrypt registered [ 2158.175150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2158.215874] alg: No test for adler32 (adler32-zlib) [ 2159.257457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.339583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2159.371298] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2160.775276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2160.827645] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2174.161292] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:09:43 (1785398983) [ 2174.446775] LNet: 73801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.449399] LNetError: 73801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2174.459639] LNet: Removed LNI 10.1.2.3@tcp [ 2174.486603] LNet: Removed LNI 192.168.202.54@tcp [ 2175.408435] Key type .llcrypt unregistered [ 2175.410064] Key type ._llcrypt unregistered [ 2185.300991] Key type ._llcrypt registered [ 2185.309832] Key type .llcrypt registered [ 2185.682802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2185.715117] alg: No test for adler32 (adler32-zlib) [ 2186.804726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.907489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2186.942512] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2188.408559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2188.448105] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2190.182870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.54@tcp [ 2191.264651] LNetError: Unexpected error -22 connecting to 192.168.202.54@tcp at host 192.168.202.54:988 [ 2198.732045] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:10:08 (1785399008) [ 2198.872602] LNet: 74932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.878464] LNetError: 74932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2198.893117] LNet: Removed LNI 10.1.2.3@tcp [ 2198.911327] LNet: Removed LNI 192.168.202.54@tcp [ 2199.630788] Key type .llcrypt unregistered [ 2199.633973] Key type ._llcrypt unregistered [ 2208.052946] Key type ._llcrypt registered [ 2208.059558] Key type .llcrypt registered [ 2208.466504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2208.512850] alg: No test for adler32 (adler32-zlib) [ 2209.648671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.787214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2209.820807] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 2211.177850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2211.205283] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 2212.807161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.54@tcp1 --nid 192.168.202.54@tcp2 [ 2220.717163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.54@tcp1 192.168.202.54@tcp2 [ 2220.801721] LNetError: 75770:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.54@tcp1 to 192.168.202.54@tcp2 - Route table may be misconfigured [ 2220.862766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.54@tcp2 192.168.202.54@tcp1 [ 2234.512591] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:10:43 (1785399043) [ 2234.740641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.794383] LNet: 76339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.815100] LNetError: 76339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.825057] LNet: Removed LNI 192.168.202.54@tcp1 [ 2235.882055] LNet: Removed LNI 192.168.202.54@tcp2 [ 2235.974473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.042503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2236.066981] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2237.091078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2237.112272] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 2238.177277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 2238.242306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.54@tcp 192.168.202.54@tcp [ 2238.309927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.54@tcp1 192.168.202.54@tcp1 [ 2238.380507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.54@tcp -d 192.168.202.54@tcp -e network_timeout [ 2238.442333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -e network_timeout [ 2238.497981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -e network_timeout [ 2238.546306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -e network_timeout [ 2238.598553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2238.625746] LNet: There was an unexpected network error while writing to 192.168.202.54: rc = -22 [ 2238.654515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2238.786495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2238.975558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2239.072260] LNet: 1 local NIs in recovery (showing 1): 192.168.202.54@tcp [ 2249.157895] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:10:57 (1785399057) [ 2249.368818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2249.447660] LNet: 77200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.456870] LNetError: 77200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.475160] LNetError: 77200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2249.498613] LNet: Removed LNI 192.168.202.54@tcp [ 2250.536864] LNet: Removed LNI 192.168.202.54@tcp1 [ 2250.609648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.775505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 2251.290411] LNet: 77244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.067767] Key type .llcrypt unregistered [ 2252.077326] Key type ._llcrypt unregistered [ 2267.255971] Key type ._llcrypt registered [ 2267.259529] Key type .llcrypt registered [ 2269.317401] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:11:18 (1785399078) [ 2270.016703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2270.047620] alg: No test for adler32 (adler32-zlib) [ 2271.016534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.748878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 2275.915092] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:11:25 (1785399085) [ 2276.783154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.859847] LNet: 78344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.969176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.133785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2277.168954] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2277.906670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2277.922605] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 2279.992711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2280.085954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp1 [ 2280.232404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp1 [ 2290.737974] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:11:40 (1785399100) [ 2292.952346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg254-server [ 2295.375671] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:11:44 (1785399104) [ 2298.032892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg254-server [ 2300.517227] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:11:49 (1785399109) [ 2303.592851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg254-server [ 2305.867197] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:11:55 (1785399115) [ 2308.646442] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg254-server [ 2310.921491] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:12:00 (1785399120) [ 2314.912647] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg254-server [ 2318.736067] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:12:06 (1785399126) [ 2321.939146] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg254-server [ 2324.348363] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:12:13 (1785399133) [ 2327.816680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg254-server [ 2331.028123] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:12:19 (1785399139) [ 2334.163664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg254-server [ 2337.515781] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:12:26 (1785399146) [ 2341.140323] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg254-server [ 2343.573522] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:12:32 (1785399152) [ 2343.712560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.738285] LNet: 82227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.743448] LNetError: 82227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.747853] LNet: Removed LNI 192.168.202.54@tcp [ 2344.807246] LNet: Removed LNI 192.168.202.54@tcp1 [ 2344.868556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.971398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2345.001039] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2346.495270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2346.775619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2347.077420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.148608] LNet: 82374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.217036] LNetError: 82374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.220373] LNetError: 82374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2347.319216] LNet: Removed LNI 192.168.202.54@tcp [ 2347.498112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.748380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2347.808325] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2349.346838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2349.686872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.738694] LNet: 82520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.756295] LNetError: 82520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.825683] LNet: Removed LNI 192.168.202.54@tcp [ 2350.883728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.032704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2351.102117] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2352.227464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2352.343709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2354.040112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.128509] LNet: 82676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.151620] LNetError: 82676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.182056] LNet: Removed LNI 192.168.202.54@tcp [ 2355.257745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.510602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2355.604797] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2356.970164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2357.383544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.488029] LNet: Removed LNI 192.168.202.54@tcp [ 2358.632091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.917297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2359.047927] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2360.984489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2361.124207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2361.595766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.673180] LNetError: 82969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.680070] LNetError: 82969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2361.708832] LNet: Removed LNI 192.168.202.54@tcp [ 2361.753975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.908868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2362.003436] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2364.445729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2366.136100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2366.220353] LNet: 83124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.222895] LNet: 83124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2367.316923] LNet: Removed LNI 192.168.202.54@tcp [ 2367.420366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.670578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2367.758129] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2369.577976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2369.783672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2371.306468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.369332] LNetError: 83281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2371.385710] LNetError: 83281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2371.434270] LNet: Removed LNI 192.168.202.54@tcp [ 2371.491844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.630650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2371.698759] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2373.174693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2373.651488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.764920] LNet: Removed LNI 192.168.202.54@tcp [ 2374.882415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.120362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2375.245046] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2376.528430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2376.719258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2377.310232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2378.407193] LNet: Removed LNI 192.168.202.54@tcp [ 2378.527644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.744632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2378.861724] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2381.335626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2381.895889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2383.023874] LNet: Removed LNI 192.168.202.54@tcp [ 2383.059858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.140591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2383.166748] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2384.067979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2384.202557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2386.064353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2386.107739] LNet: 83876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.111433] LNet: 83876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2387.175679] LNet: Removed LNI 192.168.202.54@tcp [ 2387.262858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.373524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2387.427229] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2388.730740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2388.957906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2388.974957] LNetError: 84021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.978960] LNetError: 84021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2389.993752] LNet: Removed LNI 192.168.202.54@tcp [ 2390.022324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.126940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.198448] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2391.190300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2391.228648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2391.258512] LNet: Removed LNI 192.168.202.54@tcp [ 2391.271468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2391.341473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2391.358400] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2392.020831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2392.098311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.129434] LNet: Removed LNI 192.168.202.54@tcp [ 2392.183045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.303162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2392.441878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.485670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.561878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2392.616717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.635544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.699588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2392.901082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.955280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2393.155259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2393.290847] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2395.138408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2395.232781] LNet: Removed LNI 192.168.202.54@tcp [ 2395.274786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2395.359677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2407.267131] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:13:35 (1785399215) [ 2407.949215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2408.088475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.273288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2408.372027] LNet: Added LNI 192.168.202.54@tcp231 [8/256/0/180] [ 2410.669508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.741133] LNet: Removed LNI 192.168.202.54@tcp231 [ 2410.788917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.928839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2410.999456] LNet: Added LNI 192.168.202.54@tcp231 [8/256/0/180] [ 2411.007934] LNet: 85191:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2411.964749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2412.017362] LNet: Removed LNI 192.168.202.54@tcp231 [ 2412.071113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2412.124617] LNet: Added LNI 192.168.202.54@tcp231 [8/256/0/180] [ 2414.134510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2414.211434] LNet: Removed LNI 192.168.202.54@tcp231 [ 2414.246746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2414.352728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2414.409070] LNet: Added LNI 192.168.202.54@tcp231 [8/256/0/0] [ 2414.428922] LNet: 85344:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2415.407604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2415.457357] LNet: Removed LNI 192.168.202.54@tcp231 [ 2415.505790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2415.564457] LNet: Added LNI 192.168.202.54@tcp231 [8/256/0/0] [ 2417.808325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.872702] LNet: Removed LNI 192.168.202.54@tcp231 [ 2417.904438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2418.028894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2418.078558] LNet: Added LNI 192.168.202.54@tcp231 [8/256/0/60] [ 2418.092335] LNet: 85495:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2418.791509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2418.838771] LNet: Removed LNI 192.168.202.54@tcp231 [ 2418.872521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2418.928338] LNet: Added LNI 192.168.202.54@tcp231 [8/256/0/60] [ 2421.013369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2421.062483] LNet: 85639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.070584] LNet: 85639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 2421.089492] LNetError: 85639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.092573] LNetError: 85639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2421.129157] LNet: Removed LNI 192.168.202.54@tcp231 [ 2421.189380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.287476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2421.333665] LNet: Added LNI 192.168.202.54@tcp231 [8/256/0/180] [ 2421.342325] LNet: 85647:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.092816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2422.158656] LNet: Removed LNI 192.168.202.54@tcp231 [ 2422.206614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2422.265231] LNet: Added LNI 192.168.202.54@tcp231 [8/256/0/180] [ 2434.536425] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:14:03 (1785399243) [ 2434.708740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2434.815438] LNet: Removed LNI 192.168.202.54@tcp231 [ 2434.904494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.126361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2435.202912] LNet: Added LNI 192.168.202.54@tcp232 [8/256/0/180] [ 2437.311331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2437.417558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2437.468096] LNet: Removed LNI 192.168.202.54@tcp232 [ 2437.500489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2437.569398] LNet: Added LNI 192.168.202.54@tcp232 [8/256/0/180] [ 2438.569387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2438.644655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2438.703293] LNet: Removed LNI 192.168.202.54@tcp232 [ 2438.721412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2438.773838] LNet: Added LNI 192.168.202.54@tcp232 [8/256/0/180] [ 2439.912830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2439.991188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2440.071934] LNet: Removed LNI 192.168.202.54@tcp232 [ 2440.123041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2440.188678] LNet: Added LNI 192.168.202.54@tcp232 [8/256/0/180] [ 2441.488815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2441.582827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2441.649369] LNet: Removed LNI 192.168.202.54@tcp232 [ 2441.678140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2452.226439] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 04:14:21 (1785399261) [ 2453.769798] Key type .llcrypt unregistered [ 2453.773295] Key type ._llcrypt unregistered [ 2464.140948] Key type ._llcrypt registered [ 2464.173277] Key type .llcrypt registered [ 2464.774633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2464.832229] alg: No test for adler32 (adler32-zlib) [ 2465.928577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2465.951157] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2476.774622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 2489.296878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2489.378308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2489.410084] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 2490.692167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_interrupt -r 2 [ 2490.724660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_interrupt -r 2 [ 2490.777080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_interrupt -r 2 [ 2490.854183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_interrupt -r 2 [ 2490.937706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_interrupt -r 2 [ 2491.034199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_interrupt -r 2 [ 2491.117093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_interrupt -r 2 [ 2491.189615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_interrupt -r 2 [ 2491.260615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.54@tcp 192.168.202.154@tcp [ 2491.308878] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 2491.319920] LNet: 1 local NIs in recovery (showing 1): 192.168.202.54@tcp [ 2491.673044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2491.732731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2491.776961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2491.845987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2491.909205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2491.968051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2492.047811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2492.106232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2492.126733] LNet: 88160:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2492.162721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2492.212572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2492.251809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2492.288310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2492.341461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2492.393050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2493.050623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_dropped -r 2 [ 2493.101982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_dropped -r 2 [ 2493.169150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_dropped -r 2 [ 2493.240853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_dropped -r 2 [ 2493.326450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_dropped -r 2 [ 2493.373992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_dropped -r 2 [ 2493.448915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_dropped -r 2 [ 2493.524957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_dropped -r 2 [ 2493.583786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.54@tcp 192.168.202.154@tcp [ 2493.632912] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 2494.012777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2494.045597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2494.080748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2494.109649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2494.140598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2494.182767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2494.227286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2494.273983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2494.290136] LNet: 88280:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2494.305264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2494.331330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2494.359180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2494.380727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2494.397374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2494.446912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2495.068398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_aborted -r 2 [ 2495.102388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_aborted -r 2 [ 2495.135975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_aborted -r 2 [ 2495.167145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_aborted -r 2 [ 2495.193460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_aborted -r 2 [ 2495.223801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_aborted -r 2 [ 2495.261164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_aborted -r 2 [ 2495.298704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_aborted -r 2 [ 2495.344037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.54@tcp 192.168.202.154@tcp [ 2495.388141] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 2495.745072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2495.782314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2495.825809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2495.877294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2495.922252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2495.972334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2496.021377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2496.063874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2496.083415] LNet: 88400:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2496.094855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2496.131799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2496.175864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2496.221509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2496.260244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2496.293272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2497.063653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_no_route -r 2 [ 2497.129637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_no_route -r 2 [ 2497.201868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_no_route -r 2 [ 2497.288217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_no_route -r 2 [ 2497.346922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_no_route -r 2 [ 2497.452330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_no_route -r 2 [ 2497.514870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_no_route -r 2 [ 2497.569142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_no_route -r 2 [ 2497.603030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.54@tcp 192.168.202.154@tcp [ 2497.617591] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 2497.835918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2497.883605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2497.942651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2497.996912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2498.045806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2498.098518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2498.146808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2498.193480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2498.218129] LNet: 88521:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2498.245469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2498.285362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2498.311750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2498.335259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2498.366602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2498.395882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2498.859732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e local_timeout -r 2 [ 2498.886253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e local_timeout -r 2 [ 2498.912549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e local_timeout -r 2 [ 2498.943817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e local_timeout -r 2 [ 2498.971597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e local_timeout -r 2 [ 2498.999799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e local_timeout -r 2 [ 2499.026848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e local_timeout -r 2 [ 2499.059818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e local_timeout -r 2 [ 2499.089789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.54@tcp 192.168.202.154@tcp [ 2499.435657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2499.519842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2499.629124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2499.698764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2499.746408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2499.799433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2499.858770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2499.913448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2499.970434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2500.069740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2500.112335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2500.198530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2500.268906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2500.344321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2501.250744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp -m GET -e remote_dropped -r 2 [ 2501.324645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.154@tcp1 -m GET -e remote_dropped -r 2 [ 2501.381761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp -m GET -e remote_dropped -r 2 [ 2501.451725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp -d 192.168.202.54@tcp1 -m GET -e remote_dropped -r 2 [ 2501.506487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp -m GET -e remote_dropped -r 2 [ 2501.554991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.54@tcp1 -d 192.168.202.154@tcp1 -m GET -e remote_dropped -r 2 [ 2501.616813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp -m GET -e remote_dropped -r 2 [ 2501.686310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.54@tcp1 -d 192.168.202.54@tcp1 -m GET -e remote_dropped -r 2 [ 2501.740276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.54@tcp 192.168.202.154@tcp [ 2501.776622] LNet: There was an unexpected network error while writing to 192.168.202.154: rc = -22 [ 2501.785364] LNet: Skipped 1 previous similar message [ 2501.799609] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.154@tcp [ 2502.187145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2502.272793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2502.370430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2502.421048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2502.462582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2502.503431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2502.571509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2502.661532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2502.697315] LNet: 88761:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2502.716619] LNet: 88761:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 2502.759119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2502.821836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2502.889969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2502.931614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2502.961674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2502.985630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2512.055329] LNet: 88836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2512.080741] LNetError: 88836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2513.123044] LNet: Removed LNI 192.168.202.54@tcp [ 2513.125216] LNetError: 88836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2513.151330] LNet: Removed LNI 192.168.202.54@tcp1 [ 2514.051747] Key type .llcrypt unregistered [ 2514.061434] Key type ._llcrypt unregistered [ 2540.922287] Key type ._llcrypt registered [ 2540.924840] Key type .llcrypt registered [ 2543.014340] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 04:15:51 (1785399351) [ 2543.672378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.711167] alg: No test for adler32 (adler32-zlib) [ 2544.948814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2545.055580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2545.117450] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2546.938553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2547.004416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.54@tcp [ 2547.053685] LNetError: 89816:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.54@tcp to recovery [ 2547.233698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.54@tcp [ 2547.480488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2560.985343] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 04:16:08 (1785399368) [ 2561.185559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2561.228891] LNet: 90403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2561.242009] LNetError: 90403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2561.301033] LNet: Removed LNI 192.168.202.54@tcp [ 2561.354257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2561.479492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2561.534266] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2562.921139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2562.979973] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 2564.763943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.54@tcp [ 2564.810649] LNetError: 90673:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.54@tcp to recovery [ 2565.536406] LNet: 1 local NIs in recovery (showing 1): 192.168.202.54@tcp [ 2565.557339] LNet: There was an unexpected network error while writing to 192.168.202.54: rc = -22 [ 2566.466274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.54@tcp [ 2577.741660] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 04:16:26 (1785399386) [ 2582.109303] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg254-server [ 2584.331732] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 04:16:33 (1785399393) [ 2587.312910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg254-server [ 2589.701531] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 04:16:39 (1785399399) [ 2592.437953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg254-server [ 2595.134646] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 04:16:44 (1785399404) [ 2595.395808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2595.430654] LNet: 92293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2595.440731] LNetError: 92293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2595.456206] LNet: Removed LNI 192.168.202.54@tcp [ 2595.480478] LNet: Removed LNI 192.168.202.54@tcp1 [ 2595.555165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2595.646293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2595.691399] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2597.048776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2597.197231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2597.274442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2597.439821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2597.538018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2597.641535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2610.487212] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 04:16:58 (1785399418) [ 2610.644800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2610.674159] LNet: 93017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2610.691593] LNetError: 93017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2610.706580] LNetError: 93017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2610.730708] LNet: Removed LNI 192.168.202.54@tcp [ 2610.771479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2611.013099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2611.079777] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2622.140344] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:17:11 (1785399431) [ 2622.250209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2622.279711] LNet: 93719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.288930] LNetError: 93719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.308256] LNet: Removed LNI 192.168.202.54@tcp [ 2622.329480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2622.431539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2622.461446] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2631.750254] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:17:21 (1785399441) [ 2634.258761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2637.159633] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:17:25 (1785399445) [ 2638.053156] LNet: 94761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2638.065750] LNetError: 94761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2638.086364] LNet: Removed LNI 192.168.202.54@tcp [ 2639.207820] Key type .llcrypt unregistered [ 2639.215201] Key type ._llcrypt unregistered [ 2648.503237] Key type ._llcrypt registered [ 2648.510662] Key type .llcrypt registered [ 2648.899640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2648.930154] alg: No test for adler32 (adler32-zlib) [ 2649.993101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2650.026801] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2661.554031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 2670.837074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.154@tcp [ 2671.347434] LNet: 95180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2671.366341] LNetError: 95180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2671.423745] LNet: Removed LNI 192.168.202.54@tcp [ 2672.060308] Key type .llcrypt unregistered [ 2672.065727] Key type ._llcrypt unregistered [ 2694.788270] Key type ._llcrypt registered [ 2694.790372] Key type .llcrypt registered [ 2698.574813] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2703.465453] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2708.453918] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:18:35 (1785399515) [ 2710.299964] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2712.932466] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:18:42 (1785399522) [ 2715.711000] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg254-server [ 2718.905543] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 04:18:47 (1785399527) [ 2721.589839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg254-server [ 2723.374553] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:18:53 (1785399533) [ 2725.448654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2725.643407] alg: No test for adler32 (adler32-zlib) [ 2727.082628] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 2727.583782] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2729.296850] Key type lgssc registered [ 2730.912201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2761.642939] Key type lgssc unregistered [ 2761.925338] LNet: 98360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2761.932423] LNetError: 98360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2761.956569] LNet: Removed LNI 192.168.202.54@tcp [ 2762.721445] Key type .llcrypt unregistered [ 2762.733129] Key type ._llcrypt unregistered [ 2789.937542] Key type ._llcrypt registered [ 2789.943252] Key type .llcrypt registered [ 2792.085659] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 04:20:00 (1785399600) [ 2792.865416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2792.919981] alg: No test for adler32 (adler32-zlib) [ 2793.911608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2794.014395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2794.085509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2794.152982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2794.259308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2803.789583] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 04:20:13 (1785399613) [ 2803.848078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2803.902257] LNet: 99775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2804.015227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2804.079974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2804.121367] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2804.899455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2804.915512] LNet: 99924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2804.920737] LNetError: 99924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2804.935672] LNet: Removed LNI 192.168.202.54@tcp [ 2804.948951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2805.005730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2805.022786] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2805.282432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2805.308100] LNet: Removed LNI 192.168.202.54@tcp [ 2805.319628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2805.354947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2805.370522] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2813.880624] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 04:20:23 (1785399623) [ 2814.123344] LNet: 100542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2814.133259] LNet: 100542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2814.146857] LNetError: 100542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2814.156060] LNetError: 100542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2814.176737] LNet: Removed LNI 192.168.202.54@tcp [ 2815.435239] Key type .llcrypt unregistered [ 2815.442915] Key type ._llcrypt unregistered [ 2827.493639] Key type ._llcrypt registered [ 2827.497087] Key type .llcrypt registered [ 2827.713563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2827.738126] alg: No test for adler32 (adler32-zlib) [ 2828.723461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2828.829110] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2828.830901] LNet: *** cfs_fail_loc=e003, val=100*** [ 2830.150182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2830.205845] LNet: *** cfs_fail_loc=e003, val=100*** [ 2830.212874] LNet: Skipped 2 previous similar messages [ 2830.245319] Ping reply received with mapping: [ 2830.246743] found metadata reply NI[192.168.202.54@tcp] [ 2830.248307] 36 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2830.254875] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2830.259640] 5a 5a 5a 5a 5a 5a 5a 5a [ 2830.443873] LNet: 100943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2830.450644] LNetError: 100943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2831.461662] LNet: Removed LNI 192.168.202.54@tcp [ 2832.260237] Key type .llcrypt unregistered [ 2832.265452] Key type ._llcrypt unregistered [ 2852.335065] Key type ._llcrypt registered [ 2852.340649] Key type .llcrypt registered [ 2854.899226] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 04:21:03 (1785399663) [ 2855.996852] Key type .llcrypt unregistered [ 2856.003377] Key type ._llcrypt unregistered [ 2866.422599] Key type ._llcrypt registered [ 2866.424829] Key type .llcrypt registered [ 2866.745506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2866.781044] alg: No test for adler32 (adler32-zlib) [ 2868.622145] LNetError: 101990:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2869.636751] Key type .llcrypt unregistered [ 2869.642385] Key type ._llcrypt unregistered [ 2877.473732] Key type ._llcrypt registered [ 2877.476363] Key type .llcrypt registered [ 2878.942326] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 04:21:28 (1785399688) [ 2879.574403] Key type .llcrypt unregistered [ 2879.576450] Key type ._llcrypt unregistered [ 2888.617852] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2888.623577] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2891.768518] Key type ._llcrypt registered [ 2891.783661] Key type .llcrypt registered [ 2892.311540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2892.380171] alg: No test for adler32 (adler32-zlib) [ 2893.593910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2893.724697] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2913.266465] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 04:22:01 (1785399721) [ 2913.595368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2913.652608] LNet: 103774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2913.665696] LNetError: 103774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2913.714393] LNet: Removed LNI 10.1.2.3@tcp [ 2913.752534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2915.209443] Key type .llcrypt unregistered [ 2915.212620] Key type ._llcrypt unregistered [ 2926.519698] Key type ._llcrypt registered [ 2926.521342] Key type .llcrypt registered [ 2927.005367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2927.072675] alg: No test for adler32 (adler32-zlib) [ 2928.175654] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2928.276222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 2928.541283] LNet: 104063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2928.563136] LNetError: 104063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2928.581930] LNet: Removed LNI 192.168.202.54@tcp [ 2929.929434] Key type .llcrypt unregistered [ 2929.931301] Key type ._llcrypt unregistered [ 2951.054671] Key type ._llcrypt registered [ 2951.064158] Key type .llcrypt registered [ 2953.192235] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 04:22:42 (1785399762) [ 2954.419465] Key type .llcrypt unregistered [ 2954.421549] Key type ._llcrypt unregistered [ 2965.744274] Key type ._llcrypt registered [ 2965.745290] Key type .llcrypt registered [ 2966.193168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2966.239158] alg: No test for adler32 (adler32-zlib) [ 2976.924757] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 04:23:06 (1785399786) [ 2977.274783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2977.335305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2977.436407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2977.509487] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 2979.083754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2979.159248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2979.283221] LNet: Removed LNI 192.168.202.54@tcp [ 2979.686446] LNet: 105810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2980.552390] Key type .llcrypt unregistered [ 2980.554532] Key type ._llcrypt unregistered [ 2998.292767] Key type ._llcrypt registered [ 2998.296435] Key type .llcrypt registered [ 3000.337051] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 04:23:29 (1785399809) [ 3001.077631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3001.102285] alg: No test for adler32 (adler32-zlib) [ 3002.181552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3002.342498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3002.433244] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3003.932869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3004.009012] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.54@tcp [ 3004.038208] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 3004.044174] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 3004.049314] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3003, 0, 0 [ 3004.052556] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 3004.055384] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 3004.058608] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 3004.060995] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3003, 0, 0 [ 3004.064343] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 3004.067299] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 3004.073208] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 3004.081439] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3003, 0, 3053 [ 3004.088945] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 3004.091585] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 3004.094792] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 3004.096994] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3003, 0, 3053 [ 3004.101206] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 3015.174789] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 04:23:43 (1785399823) [ 3015.627179] LNet: 107351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3015.631609] LNetError: 107351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.683668] LNet: Removed LNI 192.168.202.54@tcp [ 3017.904331] Key type .llcrypt unregistered [ 3017.905977] Key type ._llcrypt unregistered [ 3027.768298] Key type ._llcrypt registered [ 3027.778189] Key type .llcrypt registered [ 3028.281664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3028.333448] alg: No test for adler32 (adler32-zlib) [ 3029.454523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3029.526150] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3040.655357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 3052.792988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3052.909767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3052.955374] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 3054.386511] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3054.534827] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3054.542761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3054.546313] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3055.973661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 3056.016040] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 3071.516182] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 04:24:40 (1785399880) [ 3071.678384] LNet: 108847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3071.701784] LNetError: 108847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3071.704766] LNet: Removed LNI 192.168.202.54@tcp [ 3072.736262] LNet: Removed LNI 192.168.202.54@tcp1 [ 3072.745062] LNetError: 108847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3072.763630] LNetError: 108847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3072.802296] LNet: Removed LNI 10.1.2.3@tcp99 [ 3073.995419] Key type .llcrypt unregistered [ 3073.997135] Key type ._llcrypt unregistered [ 3084.058234] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3084.065276] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3087.180415] Key type ._llcrypt registered [ 3087.183909] Key type .llcrypt registered [ 3087.630608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3087.667907] alg: No test for adler32 (adler32-zlib) [ 3088.776961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3088.913900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3088.985289] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3090.473144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3090.518395] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3094.216464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 3094.308912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3094.408931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 3094.528492] LNetError: 109602:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 3094.601927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3094.719543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3095.324176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3095.442768] LNet: 109622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3095.465751] LNetError: 109622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3095.469889] LNet: Removed LNI 10.1.2.3@tcp [ 3096.487694] LNet: Removed LNI 192.168.202.54@tcp [ 3097.150152] Key type .llcrypt unregistered [ 3097.154381] Key type ._llcrypt unregistered [ 3119.432445] Key type ._llcrypt registered [ 3119.433720] Key type .llcrypt registered [ 3121.338076] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 04:25:30 (1785399930) [ 3121.878304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3121.911161] alg: No test for adler32 (adler32-zlib) [ 3122.942492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3123.013528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3123.037330] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3124.512116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3124.672953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg254-client@tcp [ 3135.028634] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 04:25:44 (1785399944) [ 3135.340447] LNet: 111156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3135.353961] LNetError: 111156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3135.420873] LNet: Removed LNI 192.168.202.54@tcp [ 3136.334503] Key type .llcrypt unregistered [ 3136.336375] Key type ._llcrypt unregistered [ 3145.621707] Key type ._llcrypt registered [ 3145.624093] Key type .llcrypt registered [ 3145.972384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3146.006741] alg: No test for adler32 (adler32-zlib) [ 3146.964853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 3147.008437] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3147.051627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 3156.071647] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 04:26:05 (1785399965) [ 3158.389821] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 3160.509441] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 04:26:10 (1785399970) [ 3163.079553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 3165.617425] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 04:26:14 (1785399974) [ 3165.991720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3166.021857] LNet: 112579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3166.032130] LNetError: 112579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3166.062371] LNet: Removed LNI 192.168.202.54@tcp [ 3166.091379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3166.168653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3166.225768] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3167.326207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3167.460293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3177.045509] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 04:26:26 (1785399986) [ 3177.416789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3177.460266] LNet: 113290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3177.471432] LNetError: 113290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3177.492164] LNet: Removed LNI 192.168.202.54@tcp [ 3177.540712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3177.622643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 3177.671966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3178.567554] Key type .llcrypt unregistered [ 3178.571159] Key type ._llcrypt unregistered [ 3196.027646] Key type ._llcrypt registered [ 3196.031756] Key type .llcrypt registered [ 3197.689659] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 04:26:46 (1785400006) [ 3198.629656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3198.692886] alg: No test for adler32 (adler32-zlib) [ 3199.767984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3199.850637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3199.878653] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3201.104710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3201.250031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3211.334913] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 04:27:00 (1785400020) [ 3211.699988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3211.751894] LNet: 114848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3211.758093] LNetError: 114848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3211.778934] LNet: Removed LNI 192.168.202.54@tcp [ 3211.832717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3211.889738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 3211.946916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 3222.942752] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 04:27:11 (1785400031) [ 3223.516846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3223.622525] LNet: 115429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3223.697757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3223.855529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3225.048218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3225.146752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3225.298423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 3226.785391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3226.861394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3227.048393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3227.097073] LNet: 115462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3227.201768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3227.357707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3228.622749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3228.711264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3228.898526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 3229.776737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3229.880586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3244.581499] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 04:27:31 (1785400051) [ 3244.893646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3245.081745] LNet: 116059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3245.214898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3245.480665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3245.560294] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3247.272510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3247.282643] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3248.027676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3248.132317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3248.587961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3248.625894] LNetError: 116344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3248.633995] LNet: Removed LNI 192.168.202.54@tcp [ 3248.660135] LNet: Removed LNI 192.168.202.54@tcp2 [ 3248.712122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3248.782580] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3248.805342] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3248.832210] LNet: 116350:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3250.040419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3250.448442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3250.474272] LNet: 116387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3250.480468] LNet: 116387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 3250.485799] LNetError: 116387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3250.494500] LNetError: 116387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3250.501619] LNet: Removed LNI 192.168.202.54@tcp [ 3250.515049] LNet: Removed LNI 192.168.202.54@tcp2 [ 3250.563190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3250.650486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3250.703548] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3252.932201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3252.954241] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3254.797847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3254.860355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3254.934776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3255.281452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3255.324607] LNetError: 116672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3255.331845] LNetError: 116672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3255.341064] LNet: Removed LNI 192.168.202.54@tcp [ 3256.366610] LNet: Removed LNI 192.168.202.54@tcp2 [ 3256.432106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3256.557736] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3256.564452] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3256.573582] LNet: 116678:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3257.512864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3257.773510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3257.804566] LNet: Removed LNI 192.168.202.54@tcp [ 3258.853905] LNet: Removed LNI 192.168.202.54@tcp2 [ 3258.935424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3259.041879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3259.076711] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3260.391443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3260.436205] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3261.736823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3261.876216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3262.241802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3262.287085] LNet: 116995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3262.299167] LNet: 116995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 3262.313698] LNetError: 116995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3262.316388] LNetError: 116995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 3262.345614] LNet: Removed LNI 192.168.202.54@tcp [ 3262.376832] LNet: Removed LNI 192.168.202.54@tcp2 [ 3262.422164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3262.529703] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3262.544613] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3262.559992] LNet: 117001:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3263.351340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3263.654880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3264.736339] LNet: Removed LNI 192.168.202.54@tcp [ 3264.762720] LNet: Removed LNI 192.168.202.54@tcp2 [ 3264.881961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3265.015052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3265.060606] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3266.063637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3266.081984] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3267.074657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3267.159291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3267.286752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3267.608461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3268.704615] LNet: Removed LNI 192.168.202.54@tcp [ 3269.763882] LNet: Removed LNI 192.168.202.54@tcp2 [ 3269.823209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3269.963528] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3269.975722] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3269.981463] LNet: 117329:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3270.807281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3271.119307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3271.131972] LNetError: 117366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3271.135496] LNetError: 117366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 3271.139506] LNet: Removed LNI 192.168.202.54@tcp [ 3272.181274] LNet: Removed LNI 192.168.202.54@tcp2 [ 3272.256333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3272.405021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3272.450710] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3273.819128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3273.859325] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3275.410271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3275.450339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3275.505220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3275.881810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3276.960315] LNet: Removed LNI 192.168.202.54@tcp [ 3276.972978] LNet: Removed LNI 192.168.202.54@tcp2 [ 3277.015759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3277.103980] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3277.109344] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3277.115932] LNet: 117669:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3277.599523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3277.963679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3279.072196] LNet: Removed LNI 192.168.202.54@tcp [ 3279.085763] LNet: Removed LNI 192.168.202.54@tcp2 [ 3279.135698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3279.250884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3279.277393] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3280.126533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3280.142813] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3281.248958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3281.309622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3281.376495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3281.469889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3282.047296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3282.086648] LNet: 118023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3282.097539] LNet: 118023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 3283.168586] LNet: Removed LNI 192.168.202.54@tcp [ 3284.200792] LNet: Removed LNI 192.168.202.54@tcp2 [ 3284.255185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3284.379774] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3284.389539] LNet: Added LNI 192.168.202.54@tcp2 [8/256/0/180] [ 3285.383260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3295.394721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 3297.807882] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 04:28:26 (1785400106) [ 3297.909556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3297.954491] LNetError: 118784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3297.963412] LNetError: 118784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 3297.973315] LNet: Removed LNI 192.168.202.54@tcp [ 3299.052455] LNet: Removed LNI 192.168.202.54@tcp2 [ 3299.102785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3299.508086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3299.529023] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3300.172905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3300.200678] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3301.081897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3301.102465] LNet: Removed LNI 10.1.2.3@tcp [ 3301.119349] LNet: Removed LNI 192.168.202.54@tcp [ 3301.150408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3301.237517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 3301.289948] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3301.307883] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3302.022509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 3302.060511] LNet: Removed LNI 192.168.202.54@tcp [ 3302.078836] LNet: Removed LNI 10.1.2.3@tcp [ 3311.227847] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 04:28:40 (1785400120) [ 3311.595794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3311.647868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3311.730549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3311.791388] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3312.555812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3312.619736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3312.714047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3312.782316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3312.825575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3312.863494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3312.917479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3312.948875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3312.992448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3313.048491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3313.126606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.54@tcp [ 3313.362407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3322.993921] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 04:28:52 (1785400132) [ 3323.330612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3323.381994] LNet: 120404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3323.389487] LNet: 120404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 3324.460040] LNet: Removed LNI 192.168.202.54@tcp [ 3324.486279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3324.628479] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3324.827880] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3324.837889] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3324.842379] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3327.276886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3327.341863] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3328.285294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3328.316571] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3332.446183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.54@tcp [ 3332.521524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 3332.678531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.54@tcp -m PUT -e local_timeout -r 1 [ 3332.850270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 3333.056338] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3333.066350] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3333.678061] LNetError: 120466:0:(lib-move.c:4801:lnet_parse()) 192.168.202.54@tcp, src 192.168.202.54@tcp: Dropping PUT (error -108 looking up sender) [ 3334.690065] LNetError: 120925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3334.710146] LNetError: 120925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 3334.746431] LNet: Removed LNI 10.1.2.3@tcp [ 3335.789144] LNet: Removed LNI 192.168.202.54@tcp [ 3336.941772] Key type .llcrypt unregistered [ 3336.943128] Key type ._llcrypt unregistered [ 3362.890036] Key type ._llcrypt registered [ 3362.891437] Key type .llcrypt registered [ 3365.000724] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 04:29:33 (1785400173) [ 3365.701694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3365.716443] alg: No test for adler32 (adler32-zlib) [ 3366.807522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3367.134284] LNet: 121781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3367.734301] Key type .llcrypt unregistered [ 3367.737684] Key type ._llcrypt unregistered [ 3376.784836] Key type ._llcrypt registered [ 3376.789730] Key type .llcrypt registered [ 3377.100515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3377.127830] alg: No test for adler32 (adler32-zlib) [ 3378.080570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3378.148532] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3389.160931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 3389.491917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 3390.342876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.154@tcp [ 3390.438510] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.154@tcp [ 3390.732375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3390.822655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3390.866554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3390.925701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3390.961633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3391.009616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3391.083757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3391.138183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3391.159907] LNet: 122232:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3391.191220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3391.236994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3391.293065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3391.342924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3391.415270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3391.474140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3392.065929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3392.321611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3392.349203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3392.372679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3392.399905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3392.427597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3392.454262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3392.476620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3392.510070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3392.528521] LNet: 122350:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3392.542784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3392.575551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3392.609753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3392.638779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3392.657927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3392.712078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3393.756672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.154@tcp [ 3394.143059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3394.223457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3394.291030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3394.364860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3394.412740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3394.467626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3394.532574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3394.604564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3394.640550] LNet: 122467:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3394.669760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3394.745638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3394.807219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3394.854610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3394.920724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3394.989888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3396.086576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3396.736111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3396.781554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3396.844173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3396.909344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3396.969808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3397.036116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3397.079971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3397.170297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3397.214719] LNet: 122584:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3397.273852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3397.370987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3397.460196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3397.540103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3397.604806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3397.674976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3398.411153] LNet: 122631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3398.449993] LNetError: 122631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3398.473707] LNet: Removed LNI 192.168.202.54@tcp [ 3399.242527] Key type .llcrypt unregistered [ 3399.246243] Key type ._llcrypt unregistered [ 3435.980489] Key type ._llcrypt registered [ 3435.981885] Key type .llcrypt registered [ 3438.529951] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 04:30:46 (1785400246) [ 3439.533917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3439.576134] alg: No test for adler32 (adler32-zlib) [ 3440.706594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3441.439662] LNet: 123537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3442.666457] Key type .llcrypt unregistered [ 3442.668056] Key type ._llcrypt unregistered [ 3454.450813] Key type ._llcrypt registered [ 3454.452994] Key type .llcrypt registered [ 3454.673692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3454.687335] alg: No test for adler32 (adler32-zlib) [ 3455.917718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3455.994538] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3477.032847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.154@tcp [ 3496.271849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3496.385915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3496.429731] LNet: Added LNI 192.168.202.54@tcp1 [8/256/0/180] [ 3498.138348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3498.875742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.154@tcp [ 3499.048525] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.154@tcp [ 3499.191892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3499.216790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3499.241758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3499.270636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3499.295740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3499.317771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3499.348230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3499.372234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3499.386916] LNet: 124195:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3499.405895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3499.428302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3499.454255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3499.481354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3499.512217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3499.539296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3500.291145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3500.577133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3500.617848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3500.655658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3500.687691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3500.732127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3500.771399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3500.818674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3500.872020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3500.914884] LNet: 124312:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3500.965868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3501.012906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3501.073620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3501.126167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3501.187791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3501.242657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3502.409807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.154@tcp [ 3502.863305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3502.915510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3503.001137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3503.066312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3503.191586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3503.272471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3503.336851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3503.426649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3503.462709] LNet: 124430:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3503.506269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3503.560158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3503.633977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3503.710280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3503.770326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3503.839167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3504.660653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3504.970173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3505.011355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3505.051559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3505.087837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3505.130838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3505.164752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3505.204108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3505.223097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3505.250915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3505.276251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3505.298575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3505.321622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3505.370400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3505.432930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3512.009647] LNet: 124628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3512.049512] LNetError: 124628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3513.057021] LNet: Removed LNI 192.168.202.54@tcp [ 3513.069528] LNetError: 124628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3513.113692] LNet: Removed LNI 192.168.202.54@tcp1 [ 3513.841962] Key type .llcrypt unregistered [ 3513.846675] Key type ._llcrypt unregistered [ 3546.616408] Key type ._llcrypt registered [ 3546.618124] Key type .llcrypt registered [ 3548.915772] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 04:32:37 (1785400357) [ 3551.200717] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3554.081803] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 04:32:43 (1785400363) [ 3565.858333] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 04:32:54 (1785400374) [ 3577.950983] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 04:33:06 (1785400386) [ 3592.195104] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 04:33:20 (1785400400) [ 3604.386727] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 04:33:33 (1785400413) [ 3618.812145] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 04:33:47 (1785400427) [ 3628.975840] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 04:33:58 (1785400438) [ 3639.381903] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 04:34:08 (1785400448) [ 3649.958109] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 04:34:19 (1785400459) [ 3659.049081] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 04:34:28 (1785400468) [ 3670.637630] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 04:34:39 (1785400479) [ 3682.397650] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 04:34:51 (1785400491) [ 3692.682000] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 04:35:01 (1785400501) [ 3702.921566] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 04:35:11 (1785400511) [ 3714.477658] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 04:35:23 (1785400523) [ 3724.164279] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 04:35:34 (1785400534) [ 3734.641360] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 04:35:43 (1785400543) [ 3743.679815] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 04:35:53 (1785400553) [ 3744.393687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3744.413974] alg: No test for adler32 (adler32-zlib) [ 3745.496398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3745.612455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3745.629860] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3746.555535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3746.587961] LNet: 135649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3746.595884] LNetError: 135649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3746.617033] LNet: Removed LNI 192.168.202.54@tcp [ 3746.639592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3746.732842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3746.784420] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3755.634790] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 04:36:05 (1785400565) [ 3766.613333] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 04:36:16 (1785400576) [ 3777.720448] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 04:36:27 (1785400587) [ 3789.046130] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 04:36:38 (1785400598) [ 3798.083663] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 04:36:47 (1785400607) [ 3807.187102] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 04:36:56 (1785400616) [ 3817.936489] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 04:37:06 (1785400626) [ 3833.816189] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 04:37:22 (1785400642) [ 3844.457571] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 04:37:33 (1785400653) [ 3859.141290] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 04:37:47 (1785400667) [ 3869.003180] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 04:37:58 (1785400678) [ 3876.909082] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 04:38:06 (1785400686) [ 3886.990931] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3590 sec ========== 04:38:15 (1785400695) [ 3887.378450] LNet: 143123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3887.387611] LNetError: 143123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3887.404223] LNet: Removed LNI 192.168.202.54@tcp [ 3888.418739] Key type .llcrypt unregistered [ 3888.420151] Key type ._llcrypt unregistered [ 3909.028816] Key type ._llcrypt registered [ 3909.114076] Key type .llcrypt registered [ 3910.010788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3910.047402] alg: No test for adler32 (adler32-zlib) [ 3911.611692] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 3912.148979] LNet: Added LNI 192.168.202.54@tcp [8/256/0/180] [ 3913.992142] Key type lgssc registered [ 3916.290221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4000.629778] Lustre: Mounted lustre-client [ 4008.893882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4022.601926] Lustre: Unmounted lustre-client [ 4057.552656] Key type lgssc unregistered [ 4057.996356] LNet: 146059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4058.005490] LNetError: 146059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4058.025295] LNet: Removed LNI 192.168.202.54@tcp [ 4059.305157] Key type .llcrypt unregistered [ 4059.306679] Key type ._llcrypt unregistered