[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 565808624 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003323] x2apic enabled [ 0.004015] Switched APIC routing to physical x2apic. [ 0.005023] kvm-guest: setup PV IPIs [ 0.007343] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.011045] LSM: Security Framework initializing [ 0.012091] Yama: becoming mindful. [ 0.013080] SELinux: Initializing. [ 0.014101] *** VALIDATE selinux *** [ 0.025346] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032023] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033207] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034143] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035163] *** VALIDATE tmpfs *** [ 0.037687] *** VALIDATE proc *** [ 0.038353] *** VALIDATE cgroup *** [ 0.039023] *** VALIDATE cgroup2 *** [ 0.040347] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041194] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043045] Spectre V2 : User space: Vulnerable [ 0.044021] Speculative Store Bypass: Vulnerable [ 0.048602] debug: unmapping init [mem 0xffffffffae059000-0xffffffffae060fff] [ 0.053636] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.055291] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.056031] ... version: 2 [ 0.057029] ... bit width: 48 [ 0.058026] ... generic registers: 4 [ 0.059028] ... value mask: 0000ffffffffffff [ 0.060033] ... max period: 00007fffffffffff [ 0.061025] ... fixed-purpose events: 3 [ 0.062025] ... event mask: 000000070000000f [ 0.064306] rcu: Hierarchical SRCU implementation. [ 0.067414] smp: Bringing up secondary CPUs ... [ 0.069103] x86: Booting SMP configuration: [ 0.070027] .... node #0, CPUs: #1 #2 #3 [ 0.097057] smp: Brought up 1 node, 4 CPUs [ 0.099021] smpboot: Max logical packages: 1 [ 0.100023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.160000] node 0 deferred pages initialised in 55ms [ 0.171338] devtmpfs: initialized [ 0.173157] x86/mm: Memory block size: 128MB [ 0.177726] gcov: version magic: 0x41383552 [ 0.181400] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.182116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.183307] pinctrl core: initialized pinctrl subsystem [ 0.184298] [ 0.185013] ************************************************************* [ 0.186022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187018] ** ** [ 0.188022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.189022] ** ** [ 0.190020] ** This means that this kernel is built to expose internal ** [ 0.191020] ** IOMMU data structures, which may compromise security on ** [ 0.192020] ** your system. ** [ 0.193023] ** ** [ 0.194024] ** If you see this message and you are not debugging the ** [ 0.195026] ** kernel, report this immediately to your vendor! ** [ 0.196026] ** ** [ 0.197019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198021] ************************************************************* [ 0.200279] NET: Registered protocol family 16 [ 0.201605] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202154] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203107] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206015] cpuidle: using governor menu [ 0.218496] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.227148] PCI: Using configuration type 1 for base access [ 0.230327] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.242173] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.243030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.245050] cryptd: max_cpu_qlen set to 1000 [ 0.246380] ACPI: Added _OSI(Module Device) [ 0.247000] ACPI: Added _OSI(Processor Device) [ 0.247015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.249020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.257111] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.268792] ACPI: Interpreter enabled [ 0.271087] ACPI: PM: (supports S0 S3 S4 S5) [ 0.273271] ACPI: Using IOAPIC for interrupt routing [ 0.279206] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.288582] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.305589] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.308067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.311025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.318112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.328000] acpiphp: Slot [2] registered [ 0.342280] acpiphp: Slot [5] registered [ 0.343000] acpiphp: Slot [6] registered [ 0.345741] acpiphp: Slot [3] registered [ 0.349178] acpiphp: Slot [4] registered [ 0.353341] acpiphp: Slot [7] registered [ 0.356231] acpiphp: Slot [8] registered [ 0.358295] acpiphp: Slot [9] registered [ 0.361181] acpiphp: Slot [10] registered [ 0.363140] acpiphp: Slot [11] registered [ 0.365208] acpiphp: Slot [12] registered [ 0.370236] acpiphp: Slot [13] registered [ 0.379493] acpiphp: Slot [14] registered [ 0.382150] acpiphp: Slot [15] registered [ 0.384133] acpiphp: Slot [16] registered [ 0.389180] acpiphp: Slot [17] registered [ 0.392166] acpiphp: Slot [18] registered [ 0.394168] acpiphp: Slot [19] registered [ 0.396156] acpiphp: Slot [20] registered [ 0.401018] acpiphp: Slot [21] registered [ 0.406187] acpiphp: Slot [22] registered [ 0.408401] acpiphp: Slot [23] registered [ 0.411226] acpiphp: Slot [24] registered [ 0.413147] acpiphp: Slot [25] registered [ 0.417844] acpiphp: Slot [26] registered [ 0.419252] acpiphp: Slot [27] registered [ 0.420169] acpiphp: Slot [28] registered [ 0.426182] acpiphp: Slot [29] registered [ 0.428312] acpiphp: Slot [30] registered [ 0.430102] acpiphp: Slot [31] registered [ 0.432131] PCI host bridge to bus 0000:00 [ 0.434031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.436032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.438033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.443032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.446034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.451034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.452191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.455961] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.464897] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.471025] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.477054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.478023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.479000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.481023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.483711] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.486795] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.489045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.492954] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.495020] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.501022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.506020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.510000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.521027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.530025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.550021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.570259] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.582021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.587000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.601021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.614000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.616562] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.619458] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.623390] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.627621] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.633176] iommu: Default domain type: Passthrough [ 0.637397] SCSI subsystem initialized [ 0.642364] ACPI: bus type USB registered [ 0.646219] usbcore: registered new interface driver usbfs [ 0.653210] usbcore: registered new interface driver hub [ 0.657138] usbcore: registered new device driver usb [ 0.663508] pps_core: LinuxPPS API ver. 1 registered [ 0.669020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.676088] PTP clock support registered [ 0.679331] EDAC MC: Ver: 3.0.0 [ 0.680000] PCI: Using ACPI for IRQ routing [ 0.682568] NetLabel: Initializing [ 0.684017] NetLabel: domain hash size = 128 [ 0.688015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.697123] NetLabel: unlabeled traffic allowed by default [ 0.705157] vgaarb: loaded [ 0.707354] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.710023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.720949] clocksource: Switched to clocksource kvm-clock [ 0.973233] VFS: Disk quotas dquot_6.6.0 [ 0.979845] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.984244] *** VALIDATE ramfs *** [ 0.992261] *** VALIDATE hugetlbfs *** [ 0.993453] pnp: PnP ACPI init [ 1.002878] pnp: PnP ACPI: found 6 devices [ 1.057567] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.071932] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.074894] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.077956] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.086514] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.095511] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.105837] NET: Registered protocol family 2 [ 1.113732] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.129883] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.138833] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.148946] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.162219] TCP: Hash tables configured (established 65536 bind 65536) [ 1.174936] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.181310] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.186322] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.193247] NET: Registered protocol family 1 [ 1.201187] RPC: Registered named UNIX socket transport module. [ 1.218573] RPC: Registered udp transport module. [ 1.220524] RPC: Registered tcp transport module. [ 1.228635] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.232933] NET: Registered protocol family 44 [ 1.235059] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.238184] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.240312] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.245281] PCI: CLS 0 bytes, default 64 [ 1.247662] Unpacking initramfs... [ 4.193382] debug: unmapping init [mem 0xffff91effcc64000-0xffff91effffcffff] [ 4.203406] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.207768] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.212319] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.735557] Initialise system trusted keyrings [ 6.738145] Key type blacklist registered [ 6.743359] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.754711] zbud: loaded [ 6.758794] *** VALIDATE nfs *** [ 6.759969] *** VALIDATE nfs4 *** [ 6.765823] pstore: using deflate compression [ 6.781983] Platform Keyring initialized [ 6.979712] NET: Registered protocol family 38 [ 6.982721] Key type asymmetric registered [ 6.985651] Asymmetric key parser 'x509' registered [ 6.988860] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.994207] io scheduler mq-deadline registered [ 6.997488] io scheduler kyber registered [ 7.000412] io scheduler bfq registered [ 7.003596] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.008730] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.013450] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.019637] ACPI: Power Button [PWRF] [ 7.033216] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.048713] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.081061] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.138950] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.191854] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.206701] Non-volatile memory driver v1.3 [ 7.213652] Linux agpgart interface v0.103 [ 7.355887] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 7.374102] vda: detected capacity change from 0 to 74838016 [ 7.432753] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.453262] vdb: detected capacity change from 0 to 1073741824 [ 7.499821] libphy: Fixed MDIO Bus: probed [ 7.511799] usbcore: registered new interface driver usbserial_generic [ 7.520326] usbserial: USB Serial support registered for generic [ 7.527282] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.537184] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.541665] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.547546] mousedev: PS/2 mouse device common for all mice [ 7.551572] rtc_cmos 00:05: RTC can wake from S4 [ 7.555391] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.559398] rtc_cmos 00:05: registered as rtc0 [ 7.562909] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.566536] intel_pstate: CPU model not supported [ 7.571089] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.572507] hid: raw HID events driver (C) Jiri Kosina [ 7.582273] usbcore: registered new interface driver usbhid [ 7.584275] usbhid: USB HID core driver [ 7.585559] drop_monitor: Initializing network drop monitor service [ 7.590640] Initializing XFRM netlink socket [ 7.592634] NET: Registered protocol family 10 [ 7.596247] Segment Routing with IPv6 [ 7.597543] NET: Registered protocol family 17 [ 7.601344] mpls_gso: MPLS GSO support [ 7.604545] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.612388] RAS: Correctable Errors collector initialized. [ 7.614863] AVX version of gcm_enc/dec engaged. [ 7.616623] AES CTR mode by8 optimization enabled [ 7.773668] sched_clock: Marking stable (7773624321, 0)->(9634205155, -1860580834) [ 7.782375] registered taskstats version 1 [ 7.786909] Loading compiled-in X.509 certificates [ 7.791425] zswap: loaded using pool lzo/zbud [ 7.838933] Key type big_key registered [ 7.863711] Key type encrypted registered [ 7.866400] ima: No TPM chip found, activating TPM-bypass! [ 7.871715] ima: Allocated hash algorithm: sha1 [ 7.875410] ima: No architecture policies found [ 7.877686] evm: Initialising EVM extended attributes: [ 7.880468] evm: security.selinux [ 7.882163] evm: security.ima [ 7.883843] evm: security.capability [ 7.885639] evm: HMAC attrs: 0x1 [ 7.890674] rtc_cmos 00:05: setting system clock to 2026-08-25 15:50:02 UTC (1787673002) [ 7.904222] debug: unmapping init [mem 0xffffffffaf003000-0xffffffffaf1fffff] [ 7.916188] debug: unmapping init [mem 0xffffffffadd82000-0xffffffffae058fff] [ 7.933532] Write protecting the kernel read-only data: 28672k [ 7.944869] debug: unmapping init [mem 0xffffffffac403000-0xffffffffac5fffff] [ 7.952796] debug: unmapping init [mem 0xffffffffacd14000-0xffffffffacdfffff] [ 8.056259] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.080519] systemd[1]: Detected virtualization kvm. [ 8.082539] systemd[1]: Detected architecture x86-64. [ 8.087829] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.155069] systemd[1]: No hostname configured. [ 8.158201] systemd[1]: Set hostname to . [ 8.161270] random: systemd: uninitialized urandom read (16 bytes read) [ 8.167433] systemd[1]: Initializing machine ID from random generator. [ 8.311875] random: ln: uninitialized urandom read (6 bytes read) [ 8.539777] random: systemd: uninitialized urandom read (16 bytes read) [ 8.543582] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 8.560279] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 8.582884] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.419614] device-mapper: uevent: version 1.0.3 [ 10.422853] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.656918] virtio_net virtio0 ens2: renamed from eth0 [ 12.958600] random: fast init done [ 13.474064] scsi host0: ata_piix [ 13.670367] scsi host1: ata_piix [ 13.675419] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 13.684595] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 18.405989] random: crng init done [ 18.411454] random: 7 urandom warning(s) missed due to ratelimiting [ 19.693962] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 21.825601] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.443719] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.339169] SELinux: Disabled at runtime. [ 25.498938] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.514979] systemd[1]: Detected virtualization kvm. [ 25.520273] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.428859] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.447905] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.468688] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.477583] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.493937] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.528967] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.579744] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Reached target Paths. [ 28.263317] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 29.148875] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.899719] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.141190] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.449857] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.563712] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 35.271375] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 36.188843] NFS: Registering the id_resolver key type [ 36.198658] Key type id_resolver registered [ 36.201011] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg302-client login: [ 93.627435] hrtimer: interrupt took 11561536 ns [ 109.855114] libcfs: loading out-of-tree module taints kernel. [ 110.011181] Key type ._llcrypt registered [ 110.013778] Key type .llcrypt registered [ 110.383779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 110.395293] alg: No test for adler32 (adler32-zlib) [ 111.862209] Lustre: Lustre: Build Version: 2.17.57_80_g62ff528 [ 113.219771] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 114.920194] Key type lgssc registered [ 117.077608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 261.398707] Lustre: Mounted lustre-client - version 2.17.57_80_g62ff528 [ 267.757725] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.199709] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing check_logdir /tmp/testlogs/ [ 287.211116] Lustre: lustre-OST0000-osc-ffff91f0476db800: disconnect after 23s idle [ 290.100720] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing yml_node [ 295.780611] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 298.643570] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 301.381837] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 303.213908] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Aug 25 11:54:56 EDT 2026 [ 321.800258] Lustre: Unmounted lustre-client [ 346.426609] Key type lgssc unregistered [ 346.854861] LNet: 8431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.864551] LNetError: 8431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 346.902292] LNet: Removed LNI 192.168.203.2@tcp [ 348.054744] Key type .llcrypt unregistered [ 348.057073] Key type ._llcrypt unregistered [ 382.369358] Key type ._llcrypt registered [ 382.374603] Key type .llcrypt registered [ 382.702667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 382.741401] alg: No test for adler32 (adler32-zlib) [ 383.781410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 383.909691] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 383.951939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 389.213472] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 389.380825] LNet: 9315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.385982] LNetError: 9315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.403831] LNet: Removed LNI 192.168.203.2@tcp [ 390.360565] Key type .llcrypt unregistered [ 390.374428] Key type ._llcrypt unregistered [ 399.906251] Key type ._llcrypt registered [ 399.915699] Key type .llcrypt registered [ 401.707257] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:56:34 (1787673394) [ 402.206880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 402.234379] alg: No test for adler32 (adler32-zlib) [ 403.304412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.453789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.501783] LNet: 9739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.563098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.672484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 403.713348] LNet: 9746:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 412.970226] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:56:45 (1787673405) [ 413.101152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.158198] LNet: 10319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.207307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.319905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 421.046985] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:56:54 (1787673414) [ 421.433404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.501508] LNet: 10912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.549160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.701943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 430.146391] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:57:03 (1787673423) [ 430.416060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.461300] LNet: 11498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.495943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.598710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 437.664333] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:57:10 (1787673430) [ 437.764357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.790955] LNet: 12073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.829944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.973189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 438.615780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 446.002798] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:57:19 (1787673439) [ 446.404233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.445873] LNet: 12672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.475753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.548401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 447.132131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 447.709897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 453.980816] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:57:27 (1787673447) [ 454.075152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.131290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.223939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 460.575634] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:57:34 (1787673454) [ 460.792534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.824856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.887662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 460.932470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 461.403957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 461.440892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 461.936100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 461.989485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 462.542111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 462.574629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 463.198069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 463.242798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 463.642164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 463.694196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 464.213727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 464.274649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 464.691710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 464.750456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 465.178633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 465.209875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 465.748487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 465.797071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 466.191817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 466.237567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 466.914537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 466.995658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 467.568559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 467.620604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 475.463824] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:57:48 (1787673468) [ 475.622774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.691764] LNet: 14530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.696654] LNet: 14530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 475.754640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.888498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 475.960286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 483.248736] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:57:56 (1787673476) [ 483.623916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.657821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.745733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 483.780553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 492.717632] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:58:05 (1787673485) [ 492.799249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.851930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.996240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 493.049844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 500.833776] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 11:58:13 (1787673493) [ 500.896612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.954205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.083681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 501.151917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 508.880453] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 11:58:21 (1787673501) [ 509.074597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.115407] LNet: 16878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 509.129316] LNet: 16878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 509.181681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.295733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 509.329639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 517.497265] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 11:58:30 (1787673510) [ 517.880472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.980457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.101041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 518.179938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 526.396840] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 11:58:39 (1787673519) [ 526.739043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.815785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.030672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 527.689513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 534.497103] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 11:58:47 (1787673527) [ 534.582575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.671716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.018664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 535.278261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 543.529866] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 11:58:56 (1787673536) [ 543.849622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.902382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.108366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 544.453935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 551.896970] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 11:59:04 (1787673544) [ 552.013915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.067345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.657411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 552.972524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 560.014786] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 11:59:13 (1787673553) [ 560.240684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.316838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.417136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 560.473512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 568.095363] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 11:59:21 (1787673561) [ 568.239157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.353364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.554536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.656358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 579.330527] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 11:59:31 (1787673571) [ 579.447609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.485409] LNet: 21731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.494167] LNet: 21731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 579.525536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.635152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 590.616215] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 11:59:43 (1787673583) [ 590.787937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.867375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.008124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 598.169543] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 11:59:51 (1787673591) [ 598.710056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.743994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.807030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 599.130878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 606.550787] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 11:59:59 (1787673599) [ 606.688529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.756676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.877892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 606.916226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 613.671554] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:00:06 (1787673606) [ 613.850955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.906162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.974932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 614.024756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 621.227776] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:00:14 (1787673614) [ 621.327711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.376256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.486179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 621.555475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 628.726835] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:00:22 (1787673622) [ 628.952969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.029830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.103635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 629.150697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 636.510853] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:00:29 (1787673629) [ 636.732497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.785501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.830583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 636.881804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 636.922845] LNetError: 25854:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 636.982862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 637.038869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 644.578127] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:00:37 (1787673637) [ 644.875561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.924459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.036898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.hFy91U9xhq [ 645.138321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.hFy91U9xhq [ 645.218261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.hFy91U9xhq [ 651.581452] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:00:44 (1787673644) [ 651.754737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.834453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.900308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 651.931520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 651.997362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 658.602335] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:00:51 (1787673651) [ 658.695715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.803679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.928634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 658.954229] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 661.022883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 662.554802] Lustre: Lustre: Build Version: 2.17.57_80_g62ff528 [ 664.560286] Key type lgssc registered [ 665.740744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 670.345486] LNetError: 27615:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.2@tcp99 has no available nets [ 673.245480] Key type lgssc unregistered [ 673.503496] LNetError: 28739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.536838] LNet: Removed LNI 192.168.203.2@tcp [ 675.205547] Key type .llcrypt unregistered [ 675.213518] Key type ._llcrypt unregistered [ 682.200973] Key type ._llcrypt registered [ 682.204548] Key type .llcrypt registered [ 683.873951] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:01:16 (1787673676) [ 684.571882] Key type .llcrypt unregistered [ 684.581265] Key type ._llcrypt unregistered [ 691.413304] Key type ._llcrypt registered [ 691.415154] Key type .llcrypt registered [ 691.702890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 691.728341] alg: No test for adler32 (adler32-zlib) [ 692.762652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.845272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 692.883267] LNet: 29660:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 692.890956] LNet: 29660:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 692.898314] LNet: 29660:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 692.920817] LNet: Message forwarding will be enabled in 2s [ 696.032248] LNet: Message forwarding enabled [ 696.546410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 696.573719] LNet: Message forwarding will stop in 2s to 6s [ 700.133705] LNet: Message forwarding disabled [ 701.227860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 701.249798] LNet: 29758:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 701.263037] LNet: 29758:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 701.266614] LNet: 29758:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 701.280386] LNet: Message forwarding will be enabled in 2s [ 701.488402] LNet: 29771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.166566] Key type .llcrypt unregistered [ 702.170411] Key type ._llcrypt unregistered [ 715.580506] Key type ._llcrypt registered [ 715.590112] Key type .llcrypt registered [ 717.264915] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:01:49 (1787673709) [ 717.727984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 717.745836] alg: No test for adler32 (adler32-zlib) [ 718.705135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.794875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 726.003956] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:01:59 (1787673719) [ 726.107608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.150800] LNet: 31172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.179481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.228985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 726.282115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 726.351250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 726.424742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 726.480728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 726.556813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 726.624549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 726.659629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 726.695626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 726.741526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 726.789277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 726.848123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 726.900220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 726.950493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 726.978470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 727.033885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 727.114826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 727.154817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 727.184198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 727.201844] LNetError: 31222:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 734.105391] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:02:07 (1787673727) [ 734.453190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.488507] LNet: 31794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.513507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.612136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 743.282762] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:02:16 (1787673736) [ 744.945624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg302-server [ 747.279682] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:02:20 (1787673740) [ 747.372621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.389454] LNet: 32720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.408923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.498163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 747.514323] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 748.317792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[3-7]@tcp [ 748.358893] LNet: 32859:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 748.421541] LNetError: 32736:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.4@tcp has gone from up to down [ 748.450264] LNetError: 32729:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.3@tcp is being used as a gateway but routing feature is not turned on [ 757.002312] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:02:30 (1787673750) [ 759.318700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg302-server [ 761.848448] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:02:34 (1787673754) [ 761.979612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.023939] LNet: 33787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.051067] LNetError: 33787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.132820] LNet: Removed LNI 192.168.203.2@tcp [ 762.219857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.342913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 762.415866] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 763.367798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[3-8/2]@tcp [ 763.391913] LNet: 33913:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 763.398712] LNet: 33913:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 763.438236] LNetError: 33799:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.3@tcp is being used as a gateway but routing feature is not turned on [ 763.444909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[3-8/2]@tcp [ 763.457949] LNetError: 33799:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 763.457976] LNetError: 33799:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.3@tcp has gone from up to down [ 763.457981] LNetError: 33799:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 771.488839] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:02:44 (1787673764) [ 771.583610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.623493] LNet: 34499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.643287] LNetError: 34499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.687174] LNet: Removed LNI 192.168.203.2@tcp [ 771.738352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.891385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 771.947556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.003423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.093642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 772.163594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.215914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.296938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 772.427762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.491803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.558426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 772.649138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.679656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.741480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 772.882105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.946719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.050678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 773.137653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.189479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.252941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 773.339306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.397974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.514240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 773.615334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.674591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.750592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 773.844505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.914882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.028688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 774.117988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.165438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.257882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 774.369775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.432880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.532484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 780.836903] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:02:54 (1787673774) [ 782.604119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg302-server [ 784.253236] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:02:57 (1787673777) [ 786.342657] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg302-server [ 787.978700] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:03:01 (1787673781) [ 788.403407] LNet: 35907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.409228] LNet: 35907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 789.465994] Key type .llcrypt unregistered [ 789.468603] Key type ._llcrypt unregistered [ 797.724317] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 798.009663] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 798.017073] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 798.019709] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 799.690648] Key type ._llcrypt registered [ 799.696463] Key type .llcrypt registered [ 799.896751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 799.918335] alg: No test for adler32 (adler32-zlib) [ 800.818375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.870929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 800.898890] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 801.657838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 801.693528] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 804.610909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 804.641726] LNet: Removed LNI 10.1.2.3@tcp [ 804.763871] LNet: 36623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.771213] LNetError: 36623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.786572] LNet: Removed LNI 192.168.203.2@tcp [ 805.567463] Key type .llcrypt unregistered [ 805.569322] Key type ._llcrypt unregistered [ 819.169729] Key type ._llcrypt registered [ 819.173929] Key type .llcrypt registered [ 820.639578] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:03:33 (1787673813) [ 821.440372] Key type .llcrypt unregistered [ 821.443426] Key type ._llcrypt unregistered [ 829.049070] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 829.209099] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 829.215596] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 829.230301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 831.922976] Key type ._llcrypt registered [ 831.924660] Key type .llcrypt registered [ 832.177736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.200768] alg: No test for adler32 (adler32-zlib) [ 833.166734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.261910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.296104] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 834.068411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 834.086922] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 836.928112] LNet: 38127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.937456] LNetError: 38127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 836.946297] LNet: Removed LNI 10.1.2.3@tcp [ 836.955890] LNet: Removed LNI 192.168.203.2@tcp [ 837.833659] Key type .llcrypt unregistered [ 837.845673] Key type ._llcrypt unregistered [ 851.067306] Key type ._llcrypt registered [ 851.070154] Key type .llcrypt registered [ 852.592574] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:04:05 (1787673845) [ 853.287766] Key type .llcrypt unregistered [ 853.289605] Key type ._llcrypt unregistered [ 860.123292] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 860.296433] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 860.301857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 860.314610] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 862.860142] Key type ._llcrypt registered [ 862.862394] Key type .llcrypt registered [ 863.034483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.053935] alg: No test for adler32 (adler32-zlib) [ 863.936444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.986559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 864.005558] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 865.692380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 865.718440] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 867.474306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 867.494831] LNet: Removed LNI 10.1.2.3@tcp [ 867.509462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 867.540242] LNet: Removed LNI 10.1.2.31@tcp [ 867.554582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 867.576830] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.238652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 868.269177] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 869.516941] LNet: 40053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.532172] LNetError: 40053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 869.541287] LNet: Removed LNI 10.1.2.31@tcp [ 869.553793] LNet: Removed LNI 10.1.2.3@tcp [ 870.320334] Key type .llcrypt unregistered [ 870.322195] Key type ._llcrypt unregistered [ 883.767793] Key type ._llcrypt registered [ 883.771886] Key type .llcrypt registered [ 884.957042] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:04:38 (1787673878) [ 885.569388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 885.609674] alg: No test for adler32 (adler32-zlib) [ 886.546659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.605685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 886.639254] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 887.391181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.419103] LNet: 41010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.424825] LNetError: 41010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.439486] LNet: Removed LNI 192.168.203.2@tcp [ 887.469714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.552304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.2@tcp [ 887.578723] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 888.199444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.2@tcp [ 888.236588] LNet: Removed LNI 192.168.203.2@tcp [ 888.309591] LNet: 41145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.190748] Key type .llcrypt unregistered [ 889.193094] Key type ._llcrypt unregistered [ 900.698991] Key type ._llcrypt registered [ 900.700669] Key type .llcrypt registered [ 901.899780] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:04:55 (1787673895) [ 903.662647] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg302-server [ 905.665389] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:04:58 (1787673898) [ 906.280427] Key type .llcrypt unregistered [ 906.283861] Key type ._llcrypt unregistered [ 912.230839] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 912.577172] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 912.593613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 912.610433] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 915.243448] Key type ._llcrypt registered [ 915.245523] Key type .llcrypt registered [ 915.449222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 915.470432] alg: No test for adler32 (adler32-zlib) [ 916.347505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.397508] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 917.019839] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 919.665823] LNet: 42991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.696180] LNetError: 42991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.714384] LNet: Removed LNI 192.168.203.2@tcp1 [ 919.748822] LNet: Removed LNI 10.1.2.3@tcp2 [ 920.712395] Key type .llcrypt unregistered [ 920.716536] Key type ._llcrypt unregistered [ 934.521134] Key type ._llcrypt registered [ 934.522955] Key type .llcrypt registered [ 935.803635] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:05:28 (1787673928) [ 936.330925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.350564] alg: No test for adler32 (adler32-zlib) [ 937.464116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.580580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 937.638119] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 939.181418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.220398] LNet: 43965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.239717] LNetError: 43965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.258680] LNet: Removed LNI 192.168.203.2@tcp [ 939.280269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.381433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 939.413426] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 948.143659] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:05:41 (1787673941) [ 948.268732] LNet: 44549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.278913] LNetError: 44549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.302635] LNet: Removed LNI 192.168.203.2@tcp [ 949.165267] Key type .llcrypt unregistered [ 949.167071] Key type ._llcrypt unregistered [ 956.431620] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 956.745567] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 956.760231] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 956.767267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 959.482379] Key type ._llcrypt registered [ 959.484983] Key type .llcrypt registered [ 959.673129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.689425] alg: No test for adler32 (adler32-zlib) [ 960.654061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.732676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.768465] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 961.560460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 961.585098] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 964.280656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.300590] LNet: 45303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.307522] LNetError: 45303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 964.317670] LNet: Removed LNI 10.1.2.3@tcp [ 964.328696] LNet: Removed LNI 192.168.203.2@tcp [ 964.354734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.446746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 964.493597] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 964.500733] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.422986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.473163] LNet: 45330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.490849] LNetError: 45330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.500771] LNetError: 45330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 965.510427] LNet: Removed LNI 10.1.2.3@tcp [ 965.527255] LNet: Removed LNI 192.168.203.2@tcp [ 965.544975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.609743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 965.675770] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 965.686149] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 966.021471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 966.044763] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 967.031986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.053989] LNet: 45492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.063511] LNetError: 45492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 967.074801] LNetError: 45492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 967.086902] LNet: Removed LNI 10.1.2.3@tcp [ 967.089569] LNet: Removed LNI 192.168.203.2@tcp [ 967.105427] LNet: Removed LNI 192.168.203.2@tcp2 [ 967.148628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.227478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 967.265476] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 967.274142] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 967.284539] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 974.776106] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:06:08 (1787673968) [ 974.894190] LNet: 46083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.900636] LNetError: 46083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 974.909414] LNetError: 46083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 974.923200] LNet: Removed LNI 10.1.2.3@tcp [ 974.931392] LNet: Removed LNI 192.168.203.2@tcp [ 974.952376] LNet: Removed LNI 192.168.203.2@tcp2 [ 975.620899] Key type .llcrypt unregistered [ 975.625616] Key type ._llcrypt unregistered [ 982.173430] Key type ._llcrypt registered [ 982.176995] Key type .llcrypt registered [ 982.420220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 982.437800] alg: No test for adler32 (adler32-zlib) [ 983.372585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.466579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 983.493578] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 984.497719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.521762] LNet: 46488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.534182] LNetError: 46488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.558587] LNet: Removed LNI 192.168.203.2@tcp [ 984.606235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.720762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 984.772422] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 985.578300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.611960] LNet: 46518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.617483] LNetError: 46518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.630860] LNet: Removed LNI 192.168.203.2@tcp [ 985.653466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.750314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 985.779964] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 986.243406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 986.283229] LNet: Removed LNI 192.168.203.2@tcp [ 986.306813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.392724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 986.436288] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 987.052090] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 987.177832] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 987.184871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 987.188588] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 988.542601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.566791] LNet: 46618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.574522] LNet: 46618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 988.581884] LNetError: 46618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.592297] LNetError: 46618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 988.606757] LNet: Removed LNI 192.168.203.2@tcp [ 988.625394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.696806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 988.733969] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 996.026154] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:06:29 (1787673989) [ 996.198924] LNet: 47208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.217777] LNetError: 47208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.265169] LNet: Removed LNI 192.168.203.2@tcp [ 997.270598] Key type .llcrypt unregistered [ 997.275707] Key type ._llcrypt unregistered [ 1004.502670] Key type ._llcrypt registered [ 1004.505602] Key type .llcrypt registered [ 1004.713822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1004.747480] alg: No test for adler32 (adler32-zlib) [ 1005.702781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.763881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1005.788881] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1006.855424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.877524] LNet: 47619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.888250] LNetError: 47619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.907491] LNet: Removed LNI 192.168.203.2@tcp [ 1006.943869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.992760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1007.013062] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1013.896549] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:06:46 (1787674006) [ 1014.099836] LNet: 48206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.125397] LNetError: 48206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.157815] LNet: Removed LNI 192.168.203.2@tcp [ 1014.767994] Key type .llcrypt unregistered [ 1014.775457] Key type ._llcrypt unregistered [ 1021.408716] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1021.527201] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1021.532669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1021.557561] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1023.243266] Key type ._llcrypt registered [ 1023.246027] Key type .llcrypt registered [ 1023.502084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1023.521282] alg: No test for adler32 (adler32-zlib) [ 1024.442684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.487784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1024.509935] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1025.323730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1025.344355] LNet: 48658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.349799] LNetError: 48658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.361685] LNet: Removed LNI 192.168.203.2@tcp [ 1025.380832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.444516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1025.474518] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1032.063645] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:07:05 (1787674025) [ 1032.290443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1032.331203] LNet: 49256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.339679] LNetError: 49256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.367493] LNet: Removed LNI 192.168.203.2@tcp [ 1032.414623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1032.585251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1039.310269] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:07:12 (1787674032) [ 1039.413232] LNet: 49830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.244706] Key type .llcrypt unregistered [ 1040.247592] Key type ._llcrypt unregistered [ 1047.075145] Key type ._llcrypt registered [ 1047.077967] Key type .llcrypt registered [ 1047.430485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1047.473700] alg: No test for adler32 (adler32-zlib) [ 1048.451249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1048.551364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1048.584983] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1049.406670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1049.433586] LNet: 50240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.438808] LNetError: 50240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.460036] LNet: Removed LNI 192.168.203.2@tcp [ 1049.488275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1049.574839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1049.622966] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1057.409655] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:07:30 (1787674050) [ 1057.525629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.570119] LNet: 50828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.588491] LNetError: 50828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.616713] LNet: Removed LNI 192.168.203.2@tcp [ 1057.670021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.775571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1064.653835] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:07:37 (1787674057) [ 1066.792680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1069.057368] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:07:42 (1787674062) [ 1071.219851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1073.251585] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:07:46 (1787674066) [ 1075.533994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1077.531537] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:07:50 (1787674070) [ 1079.448880] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1081.574162] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:07:54 (1787674074) [ 1083.713305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1085.872470] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:07:58 (1787674078) [ 1086.350562] LNet: 52889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.253803] Key type .llcrypt unregistered [ 1087.257445] Key type ._llcrypt unregistered [ 1093.091048] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1093.303402] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1093.313703] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1093.334891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1095.018940] Key type ._llcrypt registered [ 1095.023193] Key type .llcrypt registered [ 1095.427154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1095.469234] alg: No test for adler32 (adler32-zlib) [ 1096.457904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1096.601355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1096.664807] LNet: Added LNI 192.168.203.2@tcp [8/512/0/180] [ 1096.672798] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1096.679144] LNet: 53205:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1096.780231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1096.803861] LNet: 53224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.812933] LNetError: 53224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1096.823743] LNet: Removed LNI 10.1.2.3@tcp [ 1096.825391] LNetError: 53224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.858370] LNet: Removed LNI 192.168.203.2@tcp [ 1096.900634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1096.976973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1097.020952] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1097.173223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1097.239847] LNet: 53249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.254981] LNetError: 53249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.282243] LNet: Removed LNI 192.168.203.2@tcp [ 1097.317338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1097.487851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1097.548512] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1097.571515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1097.575259] LNet: 53257:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1097.825380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1097.887797] LNet: 53275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.912769] LNetError: 53275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1097.928241] LNet: Removed LNI 10.1.2.3@tcp [ 1097.936469] LNetError: 53275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.970519] LNet: Removed LNI 192.168.203.2@tcp [ 1098.009385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.104579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1098.135137] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1098.139912] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1098.148713] LNet: 53284:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1098.361084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1098.409237] LNet: 53310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.415625] LNetError: 53310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1098.430072] LNet: Removed LNI 10.1.2.3@tcp [ 1098.431899] LNetError: 53310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1098.452996] LNet: Removed LNI 192.168.203.2@tcp [ 1098.492653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.614349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1098.660397] LNet: Added LNI 192.168.203.2@tcp [8/256/0/182] [ 1098.671595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1098.887087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1098.917574] LNet: 53338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.935180] LNetError: 53338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1098.944233] LNet: Removed LNI 10.1.2.3@tcp [ 1098.947695] LNetError: 53338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1098.977332] LNet: Removed LNI 192.168.203.2@tcp [ 1099.050934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1099.155703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1099.225226] LNet: Added LNI 192.168.203.2@tcp [8/256/0/182] [ 1099.231985] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1099.237550] LNet: 53346:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1099.372485] LNet: Removed LNI 10.1.2.3@tcp [ 1099.386154] LNet: Removed LNI 192.168.203.2@tcp [ 1100.355981] Key type .llcrypt unregistered [ 1100.362982] Key type ._llcrypt unregistered [ 1113.911996] Key type ._llcrypt registered [ 1113.914428] Key type .llcrypt registered [ 1115.124874] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:08:28 (1787674108) [ 1116.853569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1118.643445] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:08:31 (1787674111) [ 1119.136512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1119.154236] alg: No test for adler32 (adler32-zlib) [ 1120.084392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1120.143900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1120.162634] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1121.074126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.2 [ 1121.100151] LNet: 54616:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1121.107536] LNet: Removed LNI 192.168.203.2@tcp [ 1128.777992] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:08:41 (1787674121) [ 1128.961192] LNet: 55179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.968542] LNetError: 55179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.002697] LNet: Removed LNI 192.168.203.2@tcp [ 1130.091546] Key type .llcrypt unregistered [ 1130.094978] Key type ._llcrypt unregistered [ 1136.636961] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1136.867717] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1139.745817] Key type ._llcrypt registered [ 1139.748768] Key type .llcrypt registered [ 1139.949162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.967475] alg: No test for adler32 (adler32-zlib) [ 1141.566873] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1148.783555] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:09:01 (1787674141) [ 1149.018976] LNet: 56104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.040810] LNetError: 56104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1149.069168] LNet: Removed LNI 10.1.2.3@tcp [ 1149.580532] Key type .llcrypt unregistered [ 1149.582494] Key type ._llcrypt unregistered [ 1155.353304] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1155.597772] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1155.600432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1155.610893] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1158.361728] Key type ._llcrypt registered [ 1158.364669] Key type .llcrypt registered [ 1158.613280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.636090] alg: No test for adler32 (adler32-zlib) [ 1159.619392] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1166.186941] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:09:19 (1787674159) [ 1167.737995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1169.585275] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:09:22 (1787674162) [ 1169.740606] LNet: 57285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.758995] LNetError: 57285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1169.783130] LNet: Removed LNI 10.1.2.3@tcp [ 1170.546763] Key type .llcrypt unregistered [ 1170.551073] Key type ._llcrypt unregistered [ 1177.432137] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1177.663535] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1177.667946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1177.686530] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1180.635819] Key type ._llcrypt registered [ 1180.637532] Key type .llcrypt registered [ 1180.922249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1180.954556] alg: No test for adler32 (adler32-zlib) [ 1182.064589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1182.105575] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1191.510456] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:09:44 (1787674184) [ 1192.007471] LNet: 58526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.011634] LNetError: 58526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1192.021803] LNet: Removed LNI 10.1.2.3@tcp [ 1192.714906] Key type .llcrypt unregistered [ 1192.718590] Key type ._llcrypt unregistered [ 1199.923702] Key type ._llcrypt registered [ 1199.925697] Key type .llcrypt registered [ 1200.244600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1200.273803] alg: No test for adler32 (adler32-zlib) [ 1201.271980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1201.331488] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1209.657718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 1210.465593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_interrupt -r 1 [ 1210.521629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_interrupt -r 1 [ 1210.558822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1210.578660] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1210.596213] LNet: 1 local NIs in recovery (showing 1): 192.168.203.2@tcp [ 1210.801897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.847361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1210.875819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1210.918912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1210.959767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1210.981753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.023419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.078111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.109321] LNet: 58968:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.131085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.189549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.244784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.293673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.345440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.410432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.454939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1212.076639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_dropped -r 1 [ 1212.134608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_dropped -r 1 [ 1212.192089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1212.225316] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1212.479807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.514725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1212.543688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.573290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.602117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.637829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.679354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.718773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.731351] LNet: 59077:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.752780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.797102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.820323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.850658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.886733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.936400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.989477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1213.534389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_aborted -r 1 [ 1213.571201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_aborted -r 1 [ 1213.593376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1213.613305] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1213.860787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1213.890551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1213.916509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.944598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.966530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.992302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.017972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.045144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.061202] LNet: 59186:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.086758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.122588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.191124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.219146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.260971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.315320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.353758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.863874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_no_route -r 1 [ 1214.893972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_no_route -r 1 [ 1214.925496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1215.158269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1215.187227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1215.236759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.272566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.296204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1215.325958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1215.351884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1215.377795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1215.416364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1215.458366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.501488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.556652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.592276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.622548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.656196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1216.286457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_timeout -r 1 [ 1216.320801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_timeout -r 1 [ 1216.354817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1216.379647] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1216.392662] LNet: Skipped 2 previous similar messages [ 1216.603795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.633405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1216.660905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.689158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.720654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.765910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.802461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.833670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.848695] LNet: 59404:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.854214] LNet: 59404:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1216.871749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.898922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.927486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.966430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1217.008545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.043382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1217.080728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1217.762944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_error -r 1 [ 1217.830710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_error -r 1 [ 1217.897478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1218.264041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1218.321282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1218.367525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.398898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.445599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.478828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.505776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.546569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.575773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.623267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.668731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.701618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.743238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.770708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.801626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1219.110514] LNet: 59559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.116910] LNetError: 59559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.134640] LNet: Removed LNI 192.168.203.2@tcp [ 1219.421433] Key type .llcrypt unregistered [ 1219.426076] Key type ._llcrypt unregistered [ 1238.878319] Key type ._llcrypt registered [ 1238.880416] Key type .llcrypt registered [ 1240.091333] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:10:33 (1787674233) [ 1240.873586] Key type .llcrypt unregistered [ 1240.877070] Key type ._llcrypt unregistered [ 1247.799618] Key type ._llcrypt registered [ 1247.806764] Key type .llcrypt registered [ 1248.142878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1248.163943] alg: No test for adler32 (adler32-zlib) [ 1249.217136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1249.245292] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1257.405204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 1268.709537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1268.753349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1268.790485] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1270.116597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_interrupt -r 1 [ 1270.163832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_interrupt -r 1 [ 1270.210979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_interrupt -r 1 [ 1270.268268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_interrupt -r 1 [ 1270.312665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_interrupt -r 1 [ 1270.371423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_interrupt -r 1 [ 1270.423971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_interrupt -r 1 [ 1270.486643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_interrupt -r 1 [ 1270.523675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1270.546634] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1270.549713] LNet: 1 local NIs in recovery (showing 1): 192.168.203.2@tcp [ 1270.840412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.899567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1270.936303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.974810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1271.075497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1271.114577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1271.174038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1271.219742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1271.246852] LNet: 61039:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1271.293664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1271.371852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1271.449534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1271.516400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1271.573925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1271.587577] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1271.596251] LNet: Skipped 3 previous similar messages [ 1271.635271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1271.681920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1272.431614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_dropped -r 1 [ 1272.477247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_dropped -r 1 [ 1272.525398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_dropped -r 1 [ 1272.570578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_dropped -r 1 [ 1272.625137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_dropped -r 1 [ 1272.676936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_dropped -r 1 [ 1272.722775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_dropped -r 1 [ 1272.807814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_dropped -r 1 [ 1272.886650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1272.945466] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1273.318032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1273.404831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1273.472129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1273.546257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1273.599167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1273.652423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1273.689572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1273.721332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1273.746167] LNet: 61161:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1273.772224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1273.835260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1273.861579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1273.880508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1273.899609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1273.920167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.943438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1274.561031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_aborted -r 1 [ 1274.594393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_aborted -r 1 [ 1274.668623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_aborted -r 1 [ 1274.736626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_aborted -r 1 [ 1274.785597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_aborted -r 1 [ 1274.843410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_aborted -r 1 [ 1274.897394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_aborted -r 1 [ 1274.943976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_aborted -r 1 [ 1274.982483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1275.026904] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1275.033253] LNet: Skipped 2 previous similar messages [ 1275.281868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.316320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1275.358673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1275.408532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.454341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1275.502537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.550599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1275.622896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1275.654118] LNet: 61282:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.680976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.749283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.832679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.883667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1275.933149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.013773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.047877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1276.640619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_no_route -r 1 [ 1276.692710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_no_route -r 1 [ 1276.739768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_no_route -r 1 [ 1276.789746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_no_route -r 1 [ 1276.829174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_no_route -r 1 [ 1276.868855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_no_route -r 1 [ 1276.900796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_no_route -r 1 [ 1276.939154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_no_route -r 1 [ 1276.984688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1277.252445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.294463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1277.359767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.421060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.474973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.519596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.577907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.635892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1277.660637] LNet: 61403:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1277.699507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1277.763572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1277.815392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.906229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.990540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.025607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.077236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1278.732491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_timeout -r 1 [ 1278.787265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_timeout -r 1 [ 1278.847335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_timeout -r 1 [ 1278.896967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_timeout -r 1 [ 1278.988228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_timeout -r 1 [ 1279.087403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_timeout -r 1 [ 1279.151988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_timeout -r 1 [ 1279.228119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_timeout -r 1 [ 1279.275264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1279.315404] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1279.337491] LNet: Skipped 5 previous similar messages [ 1279.833621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.892575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.947802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.988875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.039739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.139277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.193109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.262657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.348169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.425831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.491239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.583162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.650910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.720726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.774360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1281.441738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_error -r 1 [ 1281.474222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_error -r 1 [ 1281.516240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_error -r 1 [ 1281.569587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_error -r 1 [ 1281.626630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_error -r 1 [ 1281.709774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_error -r 1 [ 1281.770636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_error -r 1 [ 1281.825596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_error -r 1 [ 1281.912993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1282.308943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1282.397233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1282.489528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1282.539666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1282.590840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1282.638858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1282.701741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1282.774781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1282.824090] LNet: 61646:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1282.833838] LNet: 61646:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1282.878852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1282.925051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1282.962916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.009090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.042984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.091437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.133471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1289.109638] LNet: 61726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.131729] LNetError: 61726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.140276] LNet: Removed LNI 192.168.203.2@tcp [ 1290.218577] LNet: Removed LNI 192.168.203.2@tcp1 [ 1290.992039] Key type .llcrypt unregistered [ 1291.002430] Key type ._llcrypt unregistered [ 1307.805074] Key type ._llcrypt registered [ 1307.806704] Key type .llcrypt registered [ 1309.359718] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:11:42 (1787674302) [ 1310.398277] Key type .llcrypt unregistered [ 1310.400239] Key type ._llcrypt unregistered [ 1316.564917] Key type ._llcrypt registered [ 1316.567813] Key type .llcrypt registered [ 1316.795932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1316.816665] alg: No test for adler32 (adler32-zlib) [ 1317.783344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1317.850549] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1325.458647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 1326.338001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e remote_dropped -r 1 [ 1326.388945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e remote_dropped -r 1 [ 1326.446308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1326.475575] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1326.700907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1326.734127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1326.766713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1326.791282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1326.820879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1326.847427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1326.878319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1326.915568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1326.941370] LNet: 62978:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1326.975150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1327.023811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1327.049942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1327.071568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1327.092295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1327.119197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1327.152208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1327.694844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e remote_error -r 1 [ 1327.736108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e remote_error -r 1 [ 1327.795315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1327.840802] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1328.153819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1328.213417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1328.283922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1328.318838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1328.371041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1328.403141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1328.435305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1328.458658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1328.472280] LNet: 63088:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1328.488351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1328.514282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1328.539790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1328.567710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1328.614225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1328.680082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1328.720691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1329.348686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e remote_timeout -r 1 [ 1329.396839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e remote_timeout -r 1 [ 1329.463920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1329.499700] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1329.749497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1329.775115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1329.802162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1329.833719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1329.856443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1329.879078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1329.901846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1329.923668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1329.937659] LNet: 63197:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1329.951949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1329.972677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.000603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.033066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1330.064249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.094983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1330.119949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1330.408526] LNet: 63242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.412917] LNetError: 63242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.432288] LNet: Removed LNI 192.168.203.2@tcp [ 1330.813644] Key type .llcrypt unregistered [ 1330.818528] Key type ._llcrypt unregistered [ 1347.284364] Key type ._llcrypt registered [ 1347.287403] Key type .llcrypt registered [ 1348.521760] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:12:21 (1787674341) [ 1349.467652] Key type .llcrypt unregistered [ 1349.470373] Key type ._llcrypt unregistered [ 1355.923404] Key type ._llcrypt registered [ 1355.927606] Key type .llcrypt registered [ 1356.167571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1356.190769] alg: No test for adler32 (adler32-zlib) [ 1357.151425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1357.181940] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1365.138543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 1373.251990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.340656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1373.384219] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1375.313548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e remote_dropped -r 1 [ 1375.416250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e remote_dropped -r 1 [ 1375.497281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e remote_dropped -r 1 [ 1375.563124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e remote_dropped -r 1 [ 1375.629467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e remote_dropped -r 1 [ 1375.720685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e remote_dropped -r 1 [ 1375.793688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e remote_dropped -r 1 [ 1375.838249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e remote_dropped -r 1 [ 1375.922337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1375.952578] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1375.971410] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.102@tcp [ 1376.278615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1376.315915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1376.364177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1376.406828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1376.457342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1376.522720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1376.601691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1376.684443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1376.728465] LNet: 64713:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1376.761614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1376.828020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1376.863206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.898647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1376.934937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1376.960528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1376.997492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1377.035585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1377.472146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e remote_error -r 1 [ 1377.509388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e remote_error -r 1 [ 1377.546485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e remote_error -r 1 [ 1377.577640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e remote_error -r 1 [ 1377.612321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e remote_error -r 1 [ 1377.640901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e remote_error -r 1 [ 1377.685458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e remote_error -r 1 [ 1377.712539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e remote_error -r 1 [ 1377.746929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1377.784661] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1377.797544] LNet: Skipped 2 previous similar messages [ 1378.079170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1378.140747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1378.193280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1378.274940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1378.336807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1378.365636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1378.402438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1378.435176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1378.452609] LNet: 64836:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1378.474892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1378.503787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1378.546699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1378.594033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1378.632129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1378.656473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1378.711887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1378.763438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1379.324752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e remote_timeout -r 1 [ 1379.370132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e remote_timeout -r 1 [ 1379.431846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e remote_timeout -r 1 [ 1379.506555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e remote_timeout -r 1 [ 1379.591021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e remote_timeout -r 1 [ 1379.650337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e remote_timeout -r 1 [ 1379.726940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e remote_timeout -r 1 [ 1379.790779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e remote_timeout -r 1 [ 1379.861074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.102@tcp [ 1379.888983] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1379.896349] LNet: Skipped 1 previous similar message [ 1380.168776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1380.224915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1380.260824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1380.323807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1380.376073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1380.435664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1380.497297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1380.537243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1380.561108] LNet: 64959:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1380.585153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1380.618181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1380.651825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1380.704363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1380.747949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1380.789608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1380.848090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1380.891625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1386.097518] LNet: 65042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1386.111200] LNetError: 65042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1386.122430] LNet: Removed LNI 192.168.203.2@tcp [ 1386.149262] LNet: Removed LNI 192.168.203.2@tcp1 [ 1386.757770] Key type .llcrypt unregistered [ 1386.762196] Key type ._llcrypt unregistered [ 1404.027295] Key type ._llcrypt registered [ 1404.028724] Key type .llcrypt registered [ 1406.090393] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:13:18 (1787674398) [ 1408.407319] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1410.570701] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:13:23 (1787674403) [ 1412.027582] Key type .llcrypt unregistered [ 1412.030700] Key type ._llcrypt unregistered [ 1423.425930] Key type ._llcrypt registered [ 1423.430088] Key type .llcrypt registered [ 1423.730175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1423.763789] alg: No test for adler32 (adler32-zlib) [ 1424.874431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1424.923775] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1434.671173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 1435.319343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e network_timeout -r 1 [ 1435.402315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e network_timeout -r 1 [ 1435.496174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 1435.525835] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1435.821353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1435.866882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1435.907306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1435.933399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1435.954769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1435.983338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1436.001344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1436.024082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1436.039423] LNet: 66592:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1436.052717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1436.078797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1436.103843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1436.137361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1436.169265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1436.192299] LNet: 1 local NIs in recovery (showing 1): 192.168.203.2@tcp [ 1436.200026] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1436.213235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1436.419453] LNet: 66635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.426536] LNetError: 66635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.447411] LNet: Removed LNI 192.168.203.2@tcp [ 1436.824606] Key type .llcrypt unregistered [ 1436.826802] Key type ._llcrypt unregistered [ 1457.697045] Key type ._llcrypt registered [ 1457.698916] Key type .llcrypt registered [ 1458.007273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1458.037430] alg: No test for adler32 (adler32-zlib) [ 1459.048792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1459.086399] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1468.661672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 1481.623336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.708663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.748304] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1483.333454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e network_timeout -r 1 [ 1483.400174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e network_timeout -r 1 [ 1483.464633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e network_timeout -r 1 [ 1483.572531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e network_timeout -r 1 [ 1483.651485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e network_timeout -r 1 [ 1483.760741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e network_timeout -r 1 [ 1483.843706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e network_timeout -r 1 [ 1483.904405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e network_timeout -r 1 [ 1484.007706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 1484.031159] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1484.413795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1484.462560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1484.543338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1484.584545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1484.639597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1484.704140] LNet: 1 local NIs in recovery (showing 1): 192.168.203.2@tcp [ 1484.711470] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.102@tcp [ 1484.717684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1484.831943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1484.895469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1484.942608] LNet: 67522:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1484.993156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1485.049863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1485.131732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1485.191718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1485.322831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1485.405950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1493.323329] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1494.148633] LNet: 67601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.161792] LNetError: 67601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.182797] LNet: Removed LNI 192.168.203.2@tcp [ 1494.204560] LNet: Removed LNI 192.168.203.2@tcp1 [ 1495.055820] Key type .llcrypt unregistered [ 1495.060581] Key type ._llcrypt unregistered [ 1516.463143] Key type ._llcrypt registered [ 1516.465618] Key type .llcrypt registered [ 1517.679508] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:15:10 (1787674510) [ 1518.223175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1518.237368] alg: No test for adler32 (adler32-zlib) [ 1519.203045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.269307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.300550] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1519.998689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1520.015959] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1520.714596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 1520.772577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1520.823213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1520.880190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1520.954707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1521.000186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1521.038292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.2@tcp [ 1521.064335] LNetError: 68703:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.2@tcp to recovery [ 1521.312278] LNet: 1 local NIs in recovery (showing 1): 192.168.203.2@tcp [ 1521.316093] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1523.360369] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1524.562300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1527.456612] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1528.054908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1528.103395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1528.185342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.206125] LNet: 68810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.215280] LNetError: 68810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.224773] LNet: Removed LNI 192.168.203.2@tcp [ 1528.258662] LNet: Removed LNI 192.168.203.2@tcp1 [ 1528.320034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.393591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1528.423271] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1529.279940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1529.310363] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1530.137886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 1530.206903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1530.267382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1530.314565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1530.355084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1530.414720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1530.483536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1530.539465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.2@tcp [ 1530.565365] LNetError: 69083:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.2@tcp to recovery [ 1531.424644] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1534.002569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1537.570571] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1537.578059] LNet: Skipped 1 previous similar message [ 1542.114077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1542.162746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1542.213762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1549.000433] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:15:42 (1787674542) [ 1549.318940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.372179] LNet: 69801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.382731] LNetError: 69801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.395174] LNetError: 69801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1549.408672] LNet: Removed LNI 192.168.203.2@tcp [ 1549.426359] LNet: Removed LNI 192.168.203.2@tcp1 [ 1549.455022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.565507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.618324] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1550.646553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1550.664558] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1551.454840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 1551.570805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1551.625630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1551.666433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1551.725666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1551.794061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1551.864784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.2@tcp --health 0 [ 1552.014357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1552.608288] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.2@tcp [ 1552.608884] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1552.643256] LNet: Skipped 1 previous similar message [ 1567.775901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1567.950922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1568.011844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1568.071713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.2@tcp --health 500 [ 1568.196785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1576.106079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1576.161372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.180522] LNet: 70301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.186780] LNetError: 70301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.192443] LNetError: 70301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1576.198979] LNet: Removed LNI 192.168.203.2@tcp [ 1576.212864] LNet: Removed LNI 192.168.203.2@tcp1 [ 1576.226733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.265673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.289467] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1577.138574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1577.162039] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1577.906572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 1577.986633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1578.070272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1578.117407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1578.163314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1578.203225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1578.257689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1578.346119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.2@tcp --health 0 [ 1579.296610] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1579.303941] LNet: Skipped 3 previous similar messages [ 1579.819990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1590.268766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1590.308605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1590.356411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1597.068650] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:16:29 (1787674589) [ 1597.737308] LNet: 71349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.766806] LNetError: 71349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.789353] LNetError: 71349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1597.799257] LNet: Removed LNI 192.168.203.2@tcp [ 1597.820538] LNet: Removed LNI 192.168.203.2@tcp1 [ 1598.556950] Key type .llcrypt unregistered [ 1598.560760] Key type ._llcrypt unregistered [ 1605.343786] Key type ._llcrypt registered [ 1605.349850] Key type .llcrypt registered [ 1605.617391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1605.646109] alg: No test for adler32 (adler32-zlib) [ 1606.527918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1606.577964] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1614.753930] Lustre: DEBUG MARKER: Initial discovery [ 1614.855627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.102@tcp [ 1617.211209] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1617.453729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1617.530508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.102@tcp [ 1617.554742] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1617.582318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1617.632642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.102@tcp -r 1 -m GET -l 3 [ 1617.764540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.102@tcp -r 1 -m GET -e local_error [ 1617.870513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.102@tcp -r 1 -m PUT -l 6 [ 1617.888275] LNet: 1 local NIs in recovery (showing 1): 192.168.203.2@tcp [ 1619.730873] Lustre: DEBUG MARKER: Force oleg302-client.virtnet to discover 192.168.203.102@tcp (in background) [ 1620.205291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.102@tcp [ 1621.597244] Lustre: DEBUG MARKER: Force oleg302-server to discover 192.168.203.2@tcp [ 1623.329757] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1626.425402] Lustre: DEBUG MARKER: Wait for 72116 [ 1627.296266] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1628.816600] Lustre: DEBUG MARKER: Finished wait on 72116 [ 1629.550687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1629.736428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1630.204065] LNet: 72568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.208655] LNetError: 72568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1630.222338] LNet: Removed LNI 192.168.203.2@tcp [ 1630.748644] Key type .llcrypt unregistered [ 1630.752454] Key type ._llcrypt unregistered [ 1649.317497] Key type ._llcrypt registered [ 1649.319441] Key type .llcrypt registered [ 1650.533569] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:17:23 (1787674643) [ 1651.101348] Key type .llcrypt unregistered [ 1651.105626] Key type ._llcrypt unregistered [ 1657.244531] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1657.429815] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1657.433135] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1657.437673] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1660.039872] Key type ._llcrypt registered [ 1660.041728] Key type .llcrypt registered [ 1660.154589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1660.170797] alg: No test for adler32 (adler32-zlib) [ 1661.138387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.198427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.241387] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1662.154159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1662.182874] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1671.732709] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:17:44 (1787674664) [ 1671.900605] LNet: 74642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.920366] LNetError: 74642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1671.930128] LNet: Removed LNI 10.1.2.3@tcp [ 1671.962367] LNet: Removed LNI 192.168.203.2@tcp [ 1673.080620] Key type .llcrypt unregistered [ 1673.083144] Key type ._llcrypt unregistered [ 1683.731312] Key type ._llcrypt registered [ 1683.736992] Key type .llcrypt registered [ 1684.352899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1684.398306] alg: No test for adler32 (adler32-zlib) [ 1685.764651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.001852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.125409] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1688.156042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1688.214269] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1689.724973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.2@tcp [ 1690.784407] LNetError: Unexpected error -22 connecting to 192.168.203.2@tcp at host 192.168.203.2:988 [ 1697.442968] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:18:10 (1787674690) [ 1697.722273] LNet: 75786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.734090] LNetError: 75786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1697.746389] LNet: Removed LNI 10.1.2.3@tcp [ 1697.770742] LNet: Removed LNI 192.168.203.2@tcp [ 1698.819754] Key type .llcrypt unregistered [ 1698.823628] Key type ._llcrypt unregistered [ 1706.520552] Key type ._llcrypt registered [ 1706.527108] Key type .llcrypt registered [ 1706.729587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1706.743925] alg: No test for adler32 (adler32-zlib) [ 1707.677942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.733655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1707.755060] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1708.593530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1708.608559] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 1709.513410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.2@tcp1 --nid 192.168.203.2@tcp2 [ 1713.812690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.2@tcp1 192.168.203.2@tcp2 [ 1713.843730] LNetError: 76642:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.2@tcp1 to 192.168.203.2@tcp2 - Route table may be misconfigured [ 1713.879568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.2@tcp2 192.168.203.2@tcp1 [ 1719.948542] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:18:33 (1787674713) [ 1720.083321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.111792] LNet: 77211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.119665] LNetError: 77211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.124213] LNet: Removed LNI 192.168.203.2@tcp1 [ 1721.192626] LNet: Removed LNI 192.168.203.2@tcp2 [ 1721.262558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.318529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1721.350597] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1722.001774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1722.033898] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1722.862040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 1722.915836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.2@tcp 192.168.203.2@tcp [ 1722.945467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.2@tcp1 192.168.203.2@tcp1 [ 1722.984929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.2@tcp -d 192.168.203.2@tcp -e network_timeout [ 1723.030980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -e network_timeout [ 1723.088839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -e network_timeout [ 1723.131830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -e network_timeout [ 1723.177799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1723.208134] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1723.233508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1723.326855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1723.360085] LNet: 1 local NIs in recovery (showing 1): 192.168.203.2@tcp [ 1723.417618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1729.344424] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:18:42 (1787674722) [ 1729.431965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.464565] LNet: 78071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.473146] LNetError: 78071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.477621] LNetError: 78071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1730.528156] LNet: Removed LNI 192.168.203.2@tcp [ 1730.530421] LNetError: 78071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.560588] LNet: Removed LNI 192.168.203.2@tcp1 [ 1730.604366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.726814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1731.025975] LNet: 78117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.371621] Key type .llcrypt unregistered [ 1731.381506] Key type ._llcrypt unregistered [ 1742.331798] Key type ._llcrypt registered [ 1742.339988] Key type .llcrypt registered [ 1743.841445] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:18:56 (1787674736) [ 1744.205110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1744.214312] alg: No test for adler32 (adler32-zlib) [ 1745.100515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.481759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1748.530847] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:19:01 (1787674741) [ 1748.778873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.852279] LNet: 79215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.906829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.989932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.018506] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1749.626427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1749.646540] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1750.437245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1750.466724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp1 [ 1750.504576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp1 [ 1756.671194] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:19:10 (1787674750) [ 1758.401941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg302-server [ 1759.979811] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:19:13 (1787674753) [ 1761.855056] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg302-server [ 1763.476169] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:19:16 (1787674756) [ 1765.215758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg302-server [ 1766.928518] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:19:20 (1787674760) [ 1768.901500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg302-server [ 1770.817644] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:19:24 (1787674764) [ 1772.695927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg302-server [ 1774.259989] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:19:27 (1787674767) [ 1776.245838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg302-server [ 1778.278539] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:19:31 (1787674771) [ 1780.169225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg302-server [ 1782.348217] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:19:35 (1787674775) [ 1784.365047] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg302-server [ 1786.678597] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:19:39 (1787674779) [ 1788.512650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg302-server [ 1790.651301] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:19:43 (1787674783) [ 1790.924556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1790.970566] LNet: 83089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.996818] LNetError: 83089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.026911] LNet: Removed LNI 192.168.203.2@tcp [ 1792.039890] LNet: Removed LNI 192.168.203.2@tcp1 [ 1792.061495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.145696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.191389] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1793.165695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1793.282647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1793.555833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.601167] LNet: 83240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.610959] LNetError: 83240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.620568] LNetError: 83240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1793.640471] LNet: Removed LNI 192.168.203.2@tcp [ 1793.665847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.786037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1793.826729] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1794.640420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1794.937500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.992800] LNetError: 83385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.020555] LNet: Removed LNI 192.168.203.2@tcp [ 1795.074804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.186308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1795.237512] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1796.486064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1796.576718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1796.884341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.922333] LNet: 83533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.929627] LNet: 83533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1796.949229] LNet: Removed LNI 192.168.203.2@tcp [ 1796.990423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.064871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1797.106095] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1798.178945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1798.436759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.471187] LNetError: 83678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.481388] LNetError: 83678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1798.497940] LNet: Removed LNI 192.168.203.2@tcp [ 1798.524664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.674672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.724516] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1799.837542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1800.045972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1800.366802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.450593] LNet: Removed LNI 192.168.203.2@tcp [ 1801.471577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.516582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1801.545521] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1802.466671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1802.705665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.743463] LNet: 83971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1802.762353] LNet: 83971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1802.771290] LNetError: 83971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.787355] LNetError: 83971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1802.825068] LNet: Removed LNI 192.168.203.2@tcp [ 1802.857497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.923769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.951584] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1803.545086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1803.630951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1805.001256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.056128] LNet: Removed LNI 192.168.203.2@tcp [ 1806.101053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.194791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1806.230576] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1807.008428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1807.317788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.366220] LNet: Removed LNI 192.168.203.2@tcp [ 1808.418516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.485823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1808.518563] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1809.117244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1809.194426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1809.544538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1810.664633] LNet: Removed LNI 192.168.203.2@tcp [ 1810.687786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.722290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1810.735642] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1811.456621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1811.798507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.814229] LNet: 84566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.818584] LNet: 84566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1811.825068] LNetError: 84566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.831435] LNetError: 84566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1811.856322] LNet: Removed LNI 192.168.203.2@tcp [ 1811.877474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.931490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1811.949674] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1812.563644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1812.632328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1812.922779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.997353] LNet: Removed LNI 192.168.203.2@tcp [ 1814.023410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.135394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1814.161708] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1814.771335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 1815.048343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.103664] LNet: Removed LNI 192.168.203.2@tcp [ 1816.132398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.187234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1816.203952] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1817.034707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1817.127306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.169164] LNet: Removed LNI 192.168.203.2@tcp [ 1817.190954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.245158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1817.272783] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1818.023520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1818.133199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.184595] LNet: Removed LNI 192.168.203.2@tcp [ 1818.214603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.275266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1818.395981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.437627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.494416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1818.602763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.648707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.774085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1818.903765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.954963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.030811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1819.070636] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1819.920542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.966291] LNet: Removed LNI 192.168.203.2@tcp [ 1819.999524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.099458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1826.381638] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:20:19 (1787674819) [ 1826.698760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.747943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.788101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1826.809669] LNet: Added LNI 192.168.203.2@tcp231 [8/256/0/180] [ 1827.990230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.026777] LNet: 86018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.032438] LNet: 86018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1828.040061] LNetError: 86018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.050903] LNetError: 86018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1828.084953] LNet: Removed LNI 192.168.203.2@tcp231 [ 1828.126866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.243499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1828.274874] LNet: Added LNI 192.168.203.2@tcp231 [8/256/0/180] [ 1828.280522] LNet: 86026:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1828.670759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1828.707321] LNet: Removed LNI 192.168.203.2@tcp231 [ 1828.737942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1828.769742] LNet: Added LNI 192.168.203.2@tcp231 [8/256/0/180] [ 1829.908051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.943448] LNet: Removed LNI 192.168.203.2@tcp231 [ 1829.960137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.042744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1830.063101] LNet: Added LNI 192.168.203.2@tcp231 [8/256/0/0] [ 1830.075623] LNet: 86178:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1830.595700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1830.633727] LNet: Removed LNI 192.168.203.2@tcp231 [ 1830.653969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1830.691361] LNet: Added LNI 192.168.203.2@tcp231 [8/256/0/0] [ 1831.856301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.885953] LNet: Removed LNI 192.168.203.2@tcp231 [ 1831.913151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.988359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1832.010746] LNet: Added LNI 192.168.203.2@tcp231 [8/256/0/60] [ 1832.015335] LNet: 86330:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1832.392893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1832.440342] LNet: Removed LNI 192.168.203.2@tcp231 [ 1832.457116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1832.489173] LNet: Added LNI 192.168.203.2@tcp231 [8/256/0/60] [ 1833.314426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1833.351509] LNet: Removed LNI 192.168.203.2@tcp231 [ 1833.373189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.483835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1833.520815] LNet: Added LNI 192.168.203.2@tcp231 [8/256/0/180] [ 1833.987522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1834.040342] LNet: Removed LNI 192.168.203.2@tcp231 [ 1834.067483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1834.121180] LNet: Added LNI 192.168.203.2@tcp231 [8/256/0/180] [ 1843.039841] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:20:35 (1787674835) [ 1843.283991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1843.360263] LNet: Removed LNI 192.168.203.2@tcp231 [ 1843.428177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.584820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1843.612300] LNet: Added LNI 192.168.203.2@tcp232 [8/256/0/180] [ 1844.612322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1844.646256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1844.673784] LNet: Removed LNI 192.168.203.2@tcp232 [ 1844.720359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1844.738956] LNet: Added LNI 192.168.203.2@tcp232 [8/256/0/180] [ 1845.506517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1845.564131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1845.660899] LNet: Removed LNI 192.168.203.2@tcp232 [ 1845.682975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1845.734702] LNet: Added LNI 192.168.203.2@tcp232 [8/256/0/180] [ 1846.538851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1846.576216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1846.630635] LNet: Removed LNI 192.168.203.2@tcp232 [ 1846.648254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1846.678081] LNet: Added LNI 192.168.203.2@tcp232 [8/256/0/180] [ 1847.830334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1847.929480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1847.972795] LNet: Removed LNI 192.168.203.2@tcp232 [ 1847.992434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1856.318476] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:20:49 (1787674849) [ 1857.743895] Key type .llcrypt unregistered [ 1857.749788] Key type ._llcrypt unregistered [ 1868.366846] Key type ._llcrypt registered [ 1868.370112] Key type .llcrypt registered [ 1868.774446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1868.797593] alg: No test for adler32 (adler32-zlib) [ 1869.752375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1869.802783] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1877.632941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 1886.148530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.204650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1886.258548] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1887.618894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_interrupt -r 2 [ 1887.648536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_interrupt -r 2 [ 1887.672273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_interrupt -r 2 [ 1887.696888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_interrupt -r 2 [ 1887.741912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_interrupt -r 2 [ 1887.778777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_interrupt -r 2 [ 1887.817107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_interrupt -r 2 [ 1887.842461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_interrupt -r 2 [ 1887.866680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.2@tcp 192.168.203.102@tcp [ 1887.893067] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1887.904250] LNet: 1 local NIs in recovery (showing 1): 192.168.203.2@tcp [ 1888.170357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1888.198852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1888.220958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1888.250975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1888.281818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1888.314815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1888.338929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1888.363663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1888.375702] LNet: 88988:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1888.389682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1888.410933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1888.435855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1888.467538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1888.496801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1888.530618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1889.010981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_dropped -r 2 [ 1889.069633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_dropped -r 2 [ 1889.117776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_dropped -r 2 [ 1889.171947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_dropped -r 2 [ 1889.211059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_dropped -r 2 [ 1889.263625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_dropped -r 2 [ 1889.318497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_dropped -r 2 [ 1889.361551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_dropped -r 2 [ 1889.401790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.2@tcp 192.168.203.102@tcp [ 1889.421817] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1889.737235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1889.770821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1889.810541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1889.845663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1889.888650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1889.941263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1889.977951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1890.016110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1890.041783] LNet: 89108:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1890.059614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1890.105675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1890.146268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1890.190120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1890.241314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1890.282921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1890.816617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_aborted -r 2 [ 1890.847929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_aborted -r 2 [ 1890.878289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_aborted -r 2 [ 1890.909339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_aborted -r 2 [ 1890.943477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_aborted -r 2 [ 1890.974539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_aborted -r 2 [ 1891.005494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_aborted -r 2 [ 1891.043564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_aborted -r 2 [ 1891.080600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.2@tcp 192.168.203.102@tcp [ 1891.108799] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1891.295933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1891.330267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1891.361331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1891.388723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1891.422209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1891.456379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1891.488523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1891.524478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1891.544554] LNet: 89228:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1891.574718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1891.616124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1891.671609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1891.719822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1891.768608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1891.814778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1892.349050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_no_route -r 2 [ 1892.401751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_no_route -r 2 [ 1892.427282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_no_route -r 2 [ 1892.482438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_no_route -r 2 [ 1892.531732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_no_route -r 2 [ 1892.577385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_no_route -r 2 [ 1892.610817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_no_route -r 2 [ 1892.661323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_no_route -r 2 [ 1892.700863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.2@tcp 192.168.203.102@tcp [ 1892.951459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1892.983909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1893.023670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1893.048244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1893.076471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1893.117538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1893.165789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1893.207036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1893.236856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1893.284424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1893.315293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1893.349609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1893.382898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1893.423551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1893.992267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e local_timeout -r 2 [ 1894.024769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e local_timeout -r 2 [ 1894.057961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e local_timeout -r 2 [ 1894.130816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e local_timeout -r 2 [ 1894.187324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e local_timeout -r 2 [ 1894.252169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e local_timeout -r 2 [ 1894.292145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e local_timeout -r 2 [ 1894.322878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e local_timeout -r 2 [ 1894.360358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.2@tcp 192.168.203.102@tcp [ 1894.380414] LNet: There was an unexpected network error while writing to 192.168.203.102: rc = -22 [ 1894.385728] LNet: Skipped 1 previous similar message [ 1894.642657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1894.694466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1894.743793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1894.796642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1894.837859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1894.880398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1894.932196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1894.963791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1894.987871] LNet: 89469:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1894.992814] LNet: 89469:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1895.007573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1895.040429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1895.067045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1895.109852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1895.136463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1895.165986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1895.698922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp -m GET -e remote_dropped -r 2 [ 1895.756996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.102@tcp1 -m GET -e remote_dropped -r 2 [ 1895.796740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp -m GET -e remote_dropped -r 2 [ 1895.824464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp -d 192.168.203.2@tcp1 -m GET -e remote_dropped -r 2 [ 1895.868557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp -m GET -e remote_dropped -r 2 [ 1895.923212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.102@tcp1 -m GET -e remote_dropped -r 2 [ 1895.969732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp -m GET -e remote_dropped -r 2 [ 1896.003164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.2@tcp1 -d 192.168.203.2@tcp1 -m GET -e remote_dropped -r 2 [ 1896.049528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.2@tcp 192.168.203.102@tcp [ 1896.067561] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.102@tcp [ 1896.344416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1896.408491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1896.450491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1896.510477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1896.555094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1896.623226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1896.688305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1896.767828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1896.843255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1896.906574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1896.958142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1897.014396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1897.049361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1897.089223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1902.346636] LNet: 89667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.353130] LNetError: 89667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.366812] LNet: Removed LNI 192.168.203.2@tcp [ 1902.386340] LNet: Removed LNI 192.168.203.2@tcp1 [ 1902.765288] Key type .llcrypt unregistered [ 1902.771848] Key type ._llcrypt unregistered [ 1919.816675] Key type ._llcrypt registered [ 1919.823552] Key type .llcrypt registered [ 1921.227837] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:21:54 (1787674914) [ 1921.600235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1921.614801] alg: No test for adler32 (adler32-zlib) [ 1922.574063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.650876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1922.670174] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1923.434668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1923.466762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.2@tcp [ 1923.486750] LNetError: 90647:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.2@tcp to recovery [ 1923.558468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.2@tcp [ 1923.650420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1929.324296] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:22:02 (1787674922) [ 1929.645253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.681290] LNet: 91237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.695228] LNetError: 91237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.716382] LNet: Removed LNI 192.168.203.2@tcp [ 1929.740412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.811192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.849739] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1930.670936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1930.688716] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 1931.572206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.2@tcp [ 1931.588690] LNetError: 91503:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.2@tcp to recovery [ 1931.812000] LNet: 1 local NIs in recovery (showing 1): 192.168.203.2@tcp [ 1931.814754] LNet: There was an unexpected network error while writing to 192.168.203.2: rc = -22 [ 1932.899829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.2@tcp [ 1940.478754] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:22:13 (1787674933) [ 1942.776735] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg302-server [ 1944.644630] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:22:17 (1787674937) [ 1947.221352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg302-server [ 1948.922378] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:22:22 (1787674942) [ 1950.959576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg302-server [ 1953.040894] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:22:25 (1787674945) [ 1953.175061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1953.216429] LNet: 93119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.238832] LNetError: 93119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.272239] LNet: Removed LNI 192.168.203.2@tcp [ 1954.282587] LNetError: 93119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.306820] LNet: Removed LNI 192.168.203.2@tcp1 [ 1954.351237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.453516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1954.490042] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1955.444688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1955.489170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1955.525524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1955.635809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1955.726384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1955.864634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1964.112934] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:22:37 (1787674957) [ 1964.232870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.278517] LNet: 93852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.284787] LNetError: 93852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.304837] LNet: Removed LNI 192.168.203.2@tcp [ 1964.333455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.428480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1964.443991] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1971.862676] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:22:45 (1787674965) [ 1971.972963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.007502] LNet: 94555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.016390] LNetError: 94555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.034700] LNet: Removed LNI 192.168.203.2@tcp [ 1972.081037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.188768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1972.235486] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 1979.054419] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:22:52 (1787674972) [ 1980.565567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1981.915079] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:22:55 (1787674975) [ 1982.446027] LNet: 95595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.455565] LNetError: 95595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.486791] LNet: Removed LNI 192.168.203.2@tcp [ 1983.109941] Key type .llcrypt unregistered [ 1983.113457] Key type ._llcrypt unregistered [ 1990.456045] Key type ._llcrypt registered [ 1990.459425] Key type .llcrypt registered [ 1990.738258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1990.764227] alg: No test for adler32 (adler32-zlib) [ 1991.893866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1991.924920] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2000.354419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 2005.839419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.102@tcp [ 2006.398136] LNet: 96037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.425473] LNetError: 96037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.451977] LNet: Removed LNI 192.168.203.2@tcp [ 2007.029499] Key type .llcrypt unregistered [ 2007.032296] Key type ._llcrypt unregistered [ 2022.408881] Key type ._llcrypt registered [ 2022.412819] Key type .llcrypt registered [ 2023.790322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2026.541709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2027.970545] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:23:41 (1787675021) [ 2029.662559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2032.039190] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:23:44 (1787675024) [ 2034.838925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg302-server [ 2035.991736] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:23:49 (1787675029) [ 2038.682892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg302-server [ 2040.795741] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:23:53 (1787675033) [ 2042.212677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2042.242516] alg: No test for adler32 (adler32-zlib) [ 2043.364939] Lustre: Lustre: Build Version: 2.17.57_80_g62ff528 [ 2043.608028] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2045.312657] Key type lgssc registered [ 2047.795892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2073.256696] Key type lgssc unregistered [ 2073.534760] LNet: 99216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.546243] LNetError: 99216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.585250] LNet: Removed LNI 192.168.203.2@tcp [ 2074.370804] Key type .llcrypt unregistered [ 2074.377929] Key type ._llcrypt unregistered [ 2092.764262] Key type ._llcrypt registered [ 2092.767420] Key type .llcrypt registered [ 2093.983869] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:24:47 (1787675087) [ 2094.610225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2094.646446] alg: No test for adler32 (adler32-zlib) [ 2095.648531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.727797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2095.764627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2095.842833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2095.895681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2102.723673] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:24:55 (1787675095) [ 2102.859701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2102.919670] LNet: 100627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.947660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.037252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2103.061730] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2103.930063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2103.963347] LNet: 100776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.971031] LNetError: 100776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2103.996266] LNet: Removed LNI 192.168.203.2@tcp [ 2104.017753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.066645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2104.091094] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2104.590625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.644948] LNetError: 100804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.668807] LNet: Removed LNI 192.168.203.2@tcp [ 2104.715326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.793550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2104.862613] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2113.507944] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:25:06 (1787675106) [ 2113.665844] LNet: 101385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.690124] LNet: 101385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2113.717970] LNetError: 101385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.758774] LNet: Removed LNI 192.168.203.2@tcp [ 2115.008555] Key type .llcrypt unregistered [ 2115.016889] Key type ._llcrypt unregistered [ 2121.719977] Key type ._llcrypt registered [ 2121.724361] Key type .llcrypt registered [ 2122.049265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2122.079261] alg: No test for adler32 (adler32-zlib) [ 2123.003621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.108509] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2123.113936] LNet: *** cfs_fail_loc=e003, val=100*** [ 2123.841846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2123.870627] LNet: *** cfs_fail_loc=e003, val=100*** [ 2123.877088] LNet: Skipped 2 previous similar messages [ 2123.886085] Ping reply received with mapping: [ 2123.887994] found metadata reply NI[192.168.203.2@tcp] [ 2123.895854] 02 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2123.898946] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2123.902456] 5a 5a 5a 5a 5a 5a 5a 5a [ 2124.066151] LNet: 101796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.078055] LNetError: 101796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.098286] LNet: Removed LNI 192.168.203.2@tcp [ 2125.723135] Key type .llcrypt unregistered [ 2125.726536] Key type ._llcrypt unregistered [ 2136.781431] Key type ._llcrypt registered [ 2136.783979] Key type .llcrypt registered [ 2138.027199] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:25:31 (1787675131) [ 2138.760658] Key type .llcrypt unregistered [ 2138.764890] Key type ._llcrypt unregistered [ 2144.981845] Key type ._llcrypt registered [ 2144.984813] Key type .llcrypt registered [ 2145.243487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2145.277137] alg: No test for adler32 (adler32-zlib) [ 2146.370650] LNetError: 102853:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2146.654368] Key type .llcrypt unregistered [ 2146.655902] Key type ._llcrypt unregistered [ 2151.841775] Key type ._llcrypt registered [ 2151.846130] Key type .llcrypt registered [ 2153.262919] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:25:46 (1787675146) [ 2153.884544] Key type .llcrypt unregistered [ 2153.888160] Key type ._llcrypt unregistered [ 2160.878963] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2160.890221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2163.568282] Key type ._llcrypt registered [ 2163.569919] Key type .llcrypt registered [ 2163.682562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2163.689541] alg: No test for adler32 (adler32-zlib) [ 2164.693807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.812160] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2176.475926] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:26:09 (1787675169) [ 2176.842808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2176.875775] LNet: 104642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.886612] LNetError: 104642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2176.901624] LNet: Removed LNI 10.1.2.3@tcp [ 2176.931484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.818886] Key type .llcrypt unregistered [ 2177.824405] Key type ._llcrypt unregistered [ 2183.499960] Key type ._llcrypt registered [ 2183.505445] Key type .llcrypt registered [ 2183.723526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2183.744276] alg: No test for adler32 (adler32-zlib) [ 2184.658358] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2184.715636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2184.888523] LNet: 104934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.893980] LNetError: 104934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2184.917450] LNet: Removed LNI 192.168.203.2@tcp [ 2185.762698] Key type .llcrypt unregistered [ 2185.770540] Key type ._llcrypt unregistered [ 2200.089545] Key type ._llcrypt registered [ 2200.092359] Key type .llcrypt registered [ 2201.362176] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:26:34 (1787675194) [ 2202.887847] Key type .llcrypt unregistered [ 2202.899773] Key type ._llcrypt unregistered [ 2214.284641] Key type ._llcrypt registered [ 2214.289175] Key type .llcrypt registered [ 2214.641534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2214.661475] alg: No test for adler32 (adler32-zlib) [ 2225.982784] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:26:58 (1787675218) [ 2226.144827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.199189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.322559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2226.392524] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2227.434580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2227.501931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2227.555225] LNet: Removed LNI 192.168.203.2@tcp [ 2227.925886] LNet: 106691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2228.282344] Key type .llcrypt unregistered [ 2228.285968] Key type ._llcrypt unregistered [ 2239.521144] Key type ._llcrypt registered [ 2239.526490] Key type .llcrypt registered [ 2240.956214] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:27:13 (1787675233) [ 2241.574168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2241.616266] alg: No test for adler32 (adler32-zlib) [ 2242.675143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2242.731801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2242.768069] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2243.949801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2244.017414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.2@tcp [ 2244.046816] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2244.053308] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2244.062797] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2244, 0, 2294 [ 2244.072297] LNet: Conn 0 sched [nconns, cpt]: 2, 0 [ 2244.076547] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2244.082525] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2244.086645] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2244, 0, 2294 [ 2244.098963] LNet: Conn 1 sched [nconns, cpt]: 2, 0 [ 2253.135345] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:27:25 (1787675245) [ 2253.841133] LNet: 108233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.856485] LNetError: 108233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.885761] LNet: Removed LNI 192.168.203.2@tcp [ 2254.586414] Key type .llcrypt unregistered [ 2254.592200] Key type ._llcrypt unregistered [ 2263.706875] Key type ._llcrypt registered [ 2263.717489] Key type .llcrypt registered [ 2264.350187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2264.368908] alg: No test for adler32 (adler32-zlib) [ 2265.317088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2265.348981] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2273.045353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 2283.831566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.898606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2283.946505] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 2285.351477] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2285.628443] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2285.648408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2285.668279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2288.054644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2288.079899] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2300.319212] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:28:13 (1787675293) [ 2300.398662] LNet: 109734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.408135] LNetError: 109734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.414281] LNet: Removed LNI 192.168.203.2@tcp [ 2301.472159] LNet: Removed LNI 192.168.203.2@tcp1 [ 2301.475829] LNetError: 109734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2301.491663] LNetError: 109734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2301.527252] LNet: Removed LNI 10.1.2.3@tcp99 [ 2302.242676] Key type .llcrypt unregistered [ 2302.245618] Key type ._llcrypt unregistered [ 2309.767319] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2309.776454] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2312.477744] Key type ._llcrypt registered [ 2312.480211] Key type .llcrypt registered [ 2312.681048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2312.720745] alg: No test for adler32 (adler32-zlib) [ 2313.745130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2313.868259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2313.930916] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2314.916457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2314.937050] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2317.387758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2317.446337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2317.503470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2317.554133] LNetError: 110490:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2317.594408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2317.672173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2318.314634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2318.383670] LNet: 110510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.391732] LNetError: 110510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2318.397244] LNet: Removed LNI 10.1.2.3@tcp [ 2319.464718] LNet: Removed LNI 192.168.203.2@tcp [ 2320.290479] Key type .llcrypt unregistered [ 2320.294950] Key type ._llcrypt unregistered [ 2333.633646] Key type ._llcrypt registered [ 2333.634945] Key type .llcrypt registered [ 2335.241830] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:28:48 (1787675328) [ 2335.814885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.833206] alg: No test for adler32 (adler32-zlib) [ 2336.870389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.947329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2336.976890] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2337.713899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2337.806388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg302-client@tcp [ 2345.760331] Lustre: DEBUG MARKER: == sanity-lnet test 306: Concurrent lnetctl net show must not fail transiently ========================================================== 12:28:58 (1787675338) [ 2345.877780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.909567] LNet: 112043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.923525] LNetError: 112043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.984466] LNet: Removed LNI 192.168.203.2@tcp [ 2347.087647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.195735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2347.229506] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2378.461376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2384.835114] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:29:38 (1787675378) [ 2385.121890] LNet: 118368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2385.136857] LNetError: 118368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2385.157224] LNet: Removed LNI 192.168.203.2@tcp [ 2385.667854] Key type .llcrypt unregistered [ 2385.675507] Key type ._llcrypt unregistered [ 2392.783810] Key type ._llcrypt registered [ 2392.791210] Key type .llcrypt registered [ 2392.987931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2392.998838] alg: No test for adler32 (adler32-zlib) [ 2393.944388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2393.966682] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2393.994215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2400.405475] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:29:53 (1787675393) [ 2401.776496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2403.407671] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:29:56 (1787675396) [ 2404.771121] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2406.697613] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:29:59 (1787675399) [ 2406.828688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.856693] LNet: 119789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2406.865903] LNetError: 119789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2406.907241] LNet: Removed LNI 192.168.203.2@tcp [ 2406.923462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.971657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2406.986779] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2407.732925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2407.813639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2414.238598] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:30:07 (1787675407) [ 2414.311271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2414.364347] LNet: 120493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.374894] LNetError: 120493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.407736] LNet: Removed LNI 192.168.203.2@tcp [ 2414.452138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2414.594299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2414.641395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2415.481369] Key type .llcrypt unregistered [ 2415.484251] Key type ._llcrypt unregistered [ 2427.120292] Key type ._llcrypt registered [ 2427.124264] Key type .llcrypt registered [ 2428.563389] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:30:21 (1787675421) [ 2429.156850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2429.187064] alg: No test for adler32 (adler32-zlib) [ 2430.214129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2430.285663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2430.318416] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2431.176871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2431.252892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2438.586926] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:30:31 (1787675431) [ 2438.724424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2438.771628] LNet: 122056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2438.783285] LNetError: 122056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.838760] LNet: Removed LNI 192.168.203.2@tcp [ 2438.874405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.974156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2439.037635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2446.437954] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:30:39 (1787675439) [ 2446.625161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2446.669036] LNet: 122639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.717618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.786498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2447.069920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2447.089818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2447.138150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2447.500274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2447.544248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2447.658612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2447.689337] LNet: 122675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.727278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2447.810353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2448.342665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2448.379664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2448.477266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2449.042050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2449.095420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2458.213225] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:30:50 (1787675450) [ 2458.338793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2458.381629] LNet: 123270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2458.407110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2458.483517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2458.504931] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2459.316273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2459.335604] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2460.449586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2460.522856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2460.943824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2460.979696] LNetError: 123556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2460.987890] LNet: Removed LNI 192.168.203.2@tcp [ 2461.005518] LNet: Removed LNI 192.168.203.2@tcp2 [ 2461.036757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.185593] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2461.194986] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2461.212564] LNet: 123562:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2462.218866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2462.584807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2462.666410] LNet: 123599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.672398] LNet: 123599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2462.699200] LNetError: 123599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2462.708969] LNetError: 123599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2462.728567] LNet: Removed LNI 192.168.203.2@tcp [ 2462.767940] LNet: Removed LNI 192.168.203.2@tcp2 [ 2462.829760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2462.939853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2462.975714] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2463.990902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2464.031246] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2465.094373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2465.150445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2465.211040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2465.450560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2465.470437] LNetError: 123884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2465.477969] LNetError: 123884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2465.484916] LNet: Removed LNI 192.168.203.2@tcp [ 2466.540439] LNet: Removed LNI 192.168.203.2@tcp2 [ 2466.597498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2466.728481] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2466.734504] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2466.743684] LNet: 123890:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2467.172373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2467.479685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2467.513911] LNet: Removed LNI 192.168.203.2@tcp [ 2468.590308] LNet: Removed LNI 192.168.203.2@tcp2 [ 2468.648222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.761972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2468.810256] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2469.786053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2469.822301] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2471.109643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2471.146735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2471.346616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2471.371408] LNet: 124209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.376371] LNet: 124209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2471.384551] LNetError: 124209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2471.390276] LNetError: 124209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2471.396767] LNet: Removed LNI 192.168.203.2@tcp [ 2471.405916] LNet: Removed LNI 192.168.203.2@tcp2 [ 2471.425637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2471.515916] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2471.523914] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2471.537906] LNet: 124215:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2472.398876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2472.714594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2473.766342] LNet: Removed LNI 192.168.203.2@tcp [ 2473.795373] LNet: Removed LNI 192.168.203.2@tcp2 [ 2473.861523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2473.939271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2473.994517] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2475.001698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2475.025351] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2476.139483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2476.181600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2476.239750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2476.523685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2477.601329] LNet: Removed LNI 192.168.203.2@tcp [ 2478.633308] LNet: Removed LNI 192.168.203.2@tcp2 [ 2478.673330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2478.792816] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2478.806258] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2478.817145] LNet: 124543:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2479.580337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2479.802885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2479.842316] LNetError: 124580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2479.848316] LNetError: 124580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2479.854395] LNet: Removed LNI 192.168.203.2@tcp [ 2480.876431] LNet: Removed LNI 192.168.203.2@tcp2 [ 2480.933602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2481.016387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2481.043390] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2481.842642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2481.863670] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2482.631535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2482.663943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2482.764356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2483.149127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2484.192267] LNet: Removed LNI 192.168.203.2@tcp [ 2484.214044] LNet: Removed LNI 192.168.203.2@tcp2 [ 2484.283980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2484.403774] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2484.418363] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2484.426857] LNet: 124883:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2485.027830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2485.323272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2486.380061] LNet: Removed LNI 192.168.203.2@tcp [ 2486.393688] LNet: Removed LNI 192.168.203.2@tcp2 [ 2486.423560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2486.512215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2486.559672] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2487.286625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2487.303843] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2488.272212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2488.335034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2488.401470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2488.484424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2488.906411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2488.929919] LNet: 125237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.946289] LNet: 125237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2490.016670] LNet: Removed LNI 192.168.203.2@tcp [ 2491.048605] LNet: Removed LNI 192.168.203.2@tcp2 [ 2491.084449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2491.149176] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2491.164121] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2491.836123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2499.530866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2500.860996] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:31:34 (1787675494) [ 2501.126722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2501.177531] LNetError: 126002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2501.193889] LNetError: 126002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2501.206470] LNet: Removed LNI 192.168.203.2@tcp [ 2502.248297] LNet: Removed LNI 192.168.203.2@tcp2 [ 2502.320165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2502.543483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2502.563479] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2503.187965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2503.221535] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2504.396738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2504.431705] LNet: Removed LNI 10.1.2.3@tcp [ 2504.444812] LNet: Removed LNI 192.168.203.2@tcp [ 2504.461626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2504.546743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2504.585953] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2504.594652] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2505.083430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2505.112229] LNet: Removed LNI 192.168.203.2@tcp [ 2505.123314] LNet: Removed LNI 10.1.2.3@tcp [ 2513.023353] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:31:45 (1787675505) [ 2513.146958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2513.201064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2513.290302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2513.332910] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2514.080698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2514.142811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2514.202435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2514.241478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2514.311266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2514.360296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2514.437421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2514.486253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2514.525885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2514.589664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2514.652858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.2@tcp [ 2514.903662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2521.760740] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:31:55 (1787675515) [ 2522.030073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2522.075115] LNet: 127615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2522.094747] LNet: 127615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2522.130899] LNet: Removed LNI 192.168.203.2@tcp [ 2522.159884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.275427] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2522.518733] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2522.529958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2522.537996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2524.983652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2525.023200] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2526.050737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2526.094434] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2528.220158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2528.272755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2528.342155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.2@tcp -m PUT -e local_timeout -r 1 [ 2528.399583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2528.468022] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2528.475597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2528.770634] LNetError: 127681:0:(lib-move.c:4801:lnet_parse()) 192.168.203.2@tcp, src 192.168.203.2@tcp: Dropping PUT (error -108 looking up sender) [ 2529.828091] LNet: Removed LNI 10.1.2.3@tcp [ 2530.856401] LNet: Removed LNI 192.168.203.2@tcp [ 2531.317553] Key type .llcrypt unregistered [ 2531.324039] Key type ._llcrypt unregistered [ 2544.351975] Key type ._llcrypt registered [ 2544.357543] Key type .llcrypt registered [ 2546.004689] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:32:18 (1787675538) [ 2546.366878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2546.382030] alg: No test for adler32 (adler32-zlib) [ 2547.314613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2547.802498] LNet: 129012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.663359] Key type .llcrypt unregistered [ 2548.667559] Key type ._llcrypt unregistered [ 2556.340744] Key type ._llcrypt registered [ 2556.347193] Key type .llcrypt registered [ 2556.644663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2556.654164] alg: No test for adler32 (adler32-zlib) [ 2557.646961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2557.699500] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2567.162667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 2567.421859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2568.139940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.102@tcp [ 2568.413619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2568.472894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2568.516856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2568.583823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2568.621138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2568.658670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2568.710084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2568.745035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2568.782399] LNet: 129465:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2568.815530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2568.858870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2568.915702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2568.970444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2568.992136] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.102@tcp [ 2569.055759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2569.109159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2569.884874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2570.198569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2570.235868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2570.266467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2570.295151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2570.354607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2570.404887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2570.437773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2570.511303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2570.549435] LNet: 129583:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2570.579257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2570.625528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2570.665157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2570.722587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2570.807900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2570.873854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2571.654899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.102@tcp [ 2571.942564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2571.975748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2572.019838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2572.051928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2572.115631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2572.166847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2572.217939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2572.272033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2572.304990] LNet: 129700:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2572.335367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2572.395600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2572.446331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2572.485258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2572.531400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2572.564247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2573.242827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2573.477996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2573.522391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2573.574138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2573.660051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2573.687813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2573.726431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2573.756927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2573.802775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2573.844687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2573.890926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2573.943833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2574.001089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2574.068162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2574.133310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2574.676692] LNet: 129867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2574.687712] LNetError: 129867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2574.727035] LNet: Removed LNI 192.168.203.2@tcp [ 2575.149935] Key type .llcrypt unregistered [ 2575.152699] Key type ._llcrypt unregistered [ 2592.201860] Key type ._llcrypt registered [ 2592.205182] Key type .llcrypt registered [ 2593.733607] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:33:06 (1787675586) [ 2594.348420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2594.390399] alg: No test for adler32 (adler32-zlib) [ 2595.575641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2596.422374] LNet: 130756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2597.616520] Key type .llcrypt unregistered [ 2597.618719] Key type ._llcrypt unregistered [ 2605.681427] Key type ._llcrypt registered [ 2605.684276] Key type .llcrypt registered [ 2605.981828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2606.042515] alg: No test for adler32 (adler32-zlib) [ 2607.048060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2607.104663] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2615.576899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.102@tcp [ 2625.548764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2625.607139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2625.641617] LNet: Added LNI 192.168.203.2@tcp1 [8/256/0/180] [ 2626.540479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2627.089932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.102@tcp [ 2627.376455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2627.438655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2627.502495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2627.549714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2627.594645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2627.616088] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.102@tcp [ 2627.648544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2627.690440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2627.749244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2627.775648] LNet: 131413:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2627.793406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2627.825469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2627.866642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2627.914487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2627.940946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2627.969629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2628.564429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2628.808743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2628.848804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2628.882568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2628.908055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2628.937412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2628.976271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2629.012498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2629.040939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2629.061682] LNet: 131530:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2629.100597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2629.134292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2629.174720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2629.228552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2629.260338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2629.303320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2629.878253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.102@tcp [ 2630.045362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2630.079689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2630.105192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2630.129174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2630.159194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2630.198227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2630.225131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2630.286537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2630.315825] LNet: 131648:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2630.343374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2630.374789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2630.406507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2630.446362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2630.471868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2630.526700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2631.100838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2631.297131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2631.346276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2631.391022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2631.433316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2631.479542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2631.518456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2631.554686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2631.582873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2631.624106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2631.674425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2631.708787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2631.758825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2631.789210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2631.830745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2636.881086] LNet: 131849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2636.888945] LNetError: 131849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2637.921877] LNet: Removed LNI 192.168.203.2@tcp [ 2637.933214] LNetError: 131849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2637.948675] LNet: Removed LNI 192.168.203.2@tcp1 [ 2638.324976] Key type .llcrypt unregistered [ 2638.327515] Key type ._llcrypt unregistered [ 2655.119773] Key type ._llcrypt registered [ 2655.121500] Key type .llcrypt registered [ 2656.875780] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:34:09 (1787675649) [ 2657.403528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2657.434549] alg: No test for adler32 (adler32-zlib) [ 2658.336211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2658.397813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2658.418805] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2659.183382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2659.202439] LNet: Added LNI 192.168.203.2@tcp2 [8/256/0/180] [ 2660.063980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.2@tcp [ 2660.168825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2661.216702] LNet: Removed LNI 192.168.203.2@tcp [ 2668.667884] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:34:21 (1787675661) [ 2670.246399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2672.584626] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:34:25 (1787675665) [ 2681.736703] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:34:34 (1787675674) [ 2691.873267] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:34:44 (1787675684) [ 2699.670789] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:34:52 (1787675692) [ 2707.611360] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:35:00 (1787675700) [ 2716.694158] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:35:09 (1787675709) [ 2725.303592] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:35:18 (1787675718) [ 2732.408450] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:35:25 (1787675725) [ 2739.805462] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:35:33 (1787675733) [ 2747.048989] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:35:40 (1787675740) [ 2753.393874] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:35:46 (1787675746) [ 2761.124565] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:35:54 (1787675754) [ 2767.886504] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:36:01 (1787675761) [ 2774.147549] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:36:07 (1787675767) [ 2780.709377] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:36:13 (1787675773) [ 2788.396576] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:36:21 (1787675781) [ 2797.108251] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:36:30 (1787675790) [ 2805.542365] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:36:38 (1787675798) [ 2805.662184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2805.702897] LNet: 143540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2805.711877] LNetError: 143540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2805.731566] LNet: Removed LNI 192.168.203.2@tcp2 [ 2805.766979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2805.924242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2805.962804] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2807.161587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2807.183083] LNet: 143688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2807.189449] LNetError: 143688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2807.207057] LNet: Removed LNI 192.168.203.2@tcp [ 2807.227495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2807.306439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2807.330385] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2815.111828] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:36:48 (1787675808) [ 2824.610864] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:36:57 (1787675817) [ 2833.146405] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:37:05 (1787675825) [ 2843.576835] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:37:16 (1787675836) [ 2851.897941] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:37:25 (1787675845) [ 2859.820947] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:37:32 (1787675852) [ 2867.595254] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:37:40 (1787675860) [ 2875.832315] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:37:49 (1787675869) [ 2883.577516] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:37:56 (1787675876) [ 2891.808357] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:38:04 (1787675884) [ 2899.236855] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:38:12 (1787675892) [ 2906.918135] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:38:19 (1787675899) [ 2913.894155] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2609 sec ========== 12:38:27 (1787675907) [ 2914.311678] LNet: 151159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2914.326828] LNetError: 151159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2914.350352] LNet: Removed LNI 192.168.203.2@tcp [ 2914.966277] Key type .llcrypt unregistered [ 2914.971564] Key type ._llcrypt unregistered [ 2928.335392] Key type ._llcrypt registered [ 2928.375916] Key type .llcrypt registered [ 2928.957500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2928.988320] alg: No test for adler32 (adler32-zlib) [ 2930.192696] Lustre: Lustre: Build Version: 2.17.57_80_g62ff528 [ 2930.588748] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 2932.337691] Key type lgssc registered [ 2934.001502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3009.704284] Lustre: Mounted lustre-client - version 2.17.57_80_g62ff528 [ 3015.731517] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3024.564948] Lustre: Unmounted lustre-client [ 3048.925731] Key type lgssc unregistered [ 3049.169213] LNet: 154121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3049.175886] LNetError: 154121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3049.205485] LNet: Removed LNI 192.168.203.2@tcp [ 3049.883735] Key type .llcrypt unregistered [ 3049.890636] Key type ._llcrypt unregistered