[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480150498 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002357] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008003] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010118] LSM: Security Framework initializing [ 0.011047] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013056] *** VALIDATE selinux *** [ 0.021490] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026094] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] *** VALIDATE tmpfs *** [ 0.031270] *** VALIDATE proc *** [ 0.032211] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037026] Spectre V2 : User space: Vulnerable [ 0.038005] Speculative Store Bypass: Vulnerable [ 0.040830] debug: unmapping init [mem 0xffffffff8aa59000-0xffffffff8aa60fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043605] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044018] ... version: 2 [ 0.045007] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.047008] ... value mask: 0000ffffffffffff [ 0.048008] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051262] rcu: Hierarchical SRCU implementation. [ 0.053380] smp: Bringing up secondary CPUs ... [ 0.054496] x86: Booting SMP configuration: [ 0.055015] .... node #0, CPUs: #1 #2 #3 [ 0.058704] smp: Brought up 1 node, 4 CPUs [ 0.060008] smpboot: Max logical packages: 1 [ 0.061010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136319] node 0 deferred pages initialised in 71ms [ 0.140091] devtmpfs: initialized [ 0.143286] x86/mm: Memory block size: 128MB [ 0.148582] gcov: version magic: 0x41383552 [ 0.154175] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156268] pinctrl core: initialized pinctrl subsystem [ 0.157276] [ 0.157931] ************************************************************* [ 0.160014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163012] ** ** [ 0.165011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168014] ** ** [ 0.171012] ** This means that this kernel is built to expose internal ** [ 0.173012] ** IOMMU data structures, which may compromise security on ** [ 0.175012] ** your system. ** [ 0.177013] ** ** [ 0.180012] ** If you see this message and you are not debugging the ** [ 0.182010] ** kernel, report this immediately to your vendor! ** [ 0.184013] ** ** [ 0.186011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188024] ************************************************************* [ 0.191472] NET: Registered protocol family 16 [ 0.193473] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202424] cpuidle: using governor menu [ 0.204266] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207520] PCI: Using configuration type 1 for base access [ 0.209128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218197] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219132] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.221626] cryptd: max_cpu_qlen set to 1000 [ 0.224229] ACPI: Added _OSI(Module Device) [ 0.226012] ACPI: Added _OSI(Processor Device) [ 0.227012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237668] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256395] ACPI: Interpreter enabled [ 0.258170] ACPI: PM: (supports S0 S3 S4 S5) [ 0.259009] ACPI: Using IOAPIC for interrupt routing [ 0.261204] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.264784] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276729] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.280043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.284018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288186] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293380] acpiphp: Slot [2] registered [ 0.294087] acpiphp: Slot [5] registered [ 0.295079] acpiphp: Slot [6] registered [ 0.296114] acpiphp: Slot [3] registered [ 0.298063] acpiphp: Slot [4] registered [ 0.299086] acpiphp: Slot [7] registered [ 0.300104] acpiphp: Slot [8] registered [ 0.301076] acpiphp: Slot [9] registered [ 0.303073] acpiphp: Slot [10] registered [ 0.304079] acpiphp: Slot [11] registered [ 0.305063] acpiphp: Slot [12] registered [ 0.306089] acpiphp: Slot [13] registered [ 0.307066] acpiphp: Slot [14] registered [ 0.308107] acpiphp: Slot [15] registered [ 0.310071] acpiphp: Slot [16] registered [ 0.311095] acpiphp: Slot [17] registered [ 0.313161] acpiphp: Slot [18] registered [ 0.315211] acpiphp: Slot [19] registered [ 0.316111] acpiphp: Slot [20] registered [ 0.318187] acpiphp: Slot [21] registered [ 0.320194] acpiphp: Slot [22] registered [ 0.321072] acpiphp: Slot [23] registered [ 0.322302] acpiphp: Slot [24] registered [ 0.324090] acpiphp: Slot [25] registered [ 0.325077] acpiphp: Slot [26] registered [ 0.326067] acpiphp: Slot [27] registered [ 0.327062] acpiphp: Slot [28] registered [ 0.328061] acpiphp: Slot [29] registered [ 0.329062] acpiphp: Slot [30] registered [ 0.330111] acpiphp: Slot [31] registered [ 0.332043] PCI host bridge to bus 0000:00 [ 0.333011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.334015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.342015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.343194] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.345903] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.349087] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354888] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.358498] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.360024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.361017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365566] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.367035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.369653] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372823] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.384013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.388022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392085] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.403013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.452275] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.460014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498910] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517118] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.520495] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.522363] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.526775] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.531294] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.539084] iommu: Default domain type: Passthrough [ 0.541415] SCSI subsystem initialized [ 0.544149] ACPI: bus type USB registered [ 0.548137] usbcore: registered new interface driver usbfs [ 0.550065] usbcore: registered new interface driver hub [ 0.551061] usbcore: registered new device driver usb [ 0.554366] pps_core: LinuxPPS API ver. 1 registered [ 0.557010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.559062] PTP clock support registered [ 0.562033] EDAC MC: Ver: 3.0.0 [ 0.564108] PCI: Using ACPI for IRQ routing [ 0.565688] NetLabel: Initializing [ 0.566007] NetLabel: domain hash size = 128 [ 0.567008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.568098] NetLabel: unlabeled traffic allowed by default [ 0.570073] vgaarb: loaded [ 0.572029] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.574010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.581069] clocksource: Switched to clocksource kvm-clock [ 0.721452] VFS: Disk quotas dquot_6.6.0 [ 0.722816] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.725312] *** VALIDATE ramfs *** [ 0.727584] *** VALIDATE hugetlbfs *** [ 0.732333] pnp: PnP ACPI init [ 0.735189] pnp: PnP ACPI: found 6 devices [ 0.769865] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.772458] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.774632] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.776375] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.778282] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.780242] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.782561] NET: Registered protocol family 2 [ 0.784568] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.788773] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.792442] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.797400] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.800616] TCP: Hash tables configured (established 65536 bind 65536) [ 0.803267] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.806020] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.808605] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.811313] NET: Registered protocol family 1 [ 0.814233] RPC: Registered named UNIX socket transport module. [ 0.816363] RPC: Registered udp transport module. [ 0.817744] RPC: Registered tcp transport module. [ 0.819307] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.821377] NET: Registered protocol family 44 [ 0.822727] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.824676] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.826636] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.828753] PCI: CLS 0 bytes, default 64 [ 0.832541] Unpacking initramfs... [ 2.306630] debug: unmapping init [mem 0xffff9eaafcc64000-0xffff9eaafffcffff] [ 2.310943] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.313204] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.316072] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.792204] Initialise system trusted keyrings [ 2.793811] Key type blacklist registered [ 2.795723] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.804839] zbud: loaded [ 2.807763] *** VALIDATE nfs *** [ 2.808901] *** VALIDATE nfs4 *** [ 2.810385] pstore: using deflate compression [ 2.814140] Platform Keyring initialized [ 2.915210] NET: Registered protocol family 38 [ 2.916654] Key type asymmetric registered [ 2.917852] Asymmetric key parser 'x509' registered [ 2.919390] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.921981] io scheduler mq-deadline registered [ 2.923305] io scheduler kyber registered [ 2.924508] io scheduler bfq registered [ 2.925968] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.928550] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.930974] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.933575] ACPI: Power Button [PWRF] [ 2.938134] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.948737] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.959082] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.992484] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.028657] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.034663] Non-volatile memory driver v1.3 [ 3.036150] Linux agpgart interface v0.103 [ 3.062164] virtio_blk virtio1: [vda] 150064 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.064764] vda: detected capacity change from 0 to 76832768 [ 3.084705] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.087497] vdb: detected capacity change from 0 to 1073741824 [ 3.101162] libphy: Fixed MDIO Bus: probed [ 3.105992] usbcore: registered new interface driver usbserial_generic [ 3.108160] usbserial: USB Serial support registered for generic [ 3.110244] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.114273] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.115929] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.118214] mousedev: PS/2 mouse device common for all mice [ 3.121152] rtc_cmos 00:05: RTC can wake from S4 [ 3.125085] rtc_cmos 00:05: registered as rtc0 [ 3.127127] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.130164] intel_pstate: CPU model not supported [ 3.133025] hid: raw HID events driver (C) Jiri Kosina [ 3.135140] usbcore: registered new interface driver usbhid [ 3.135156] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.136561] usbhid: USB HID core driver [ 3.136690] drop_monitor: Initializing network drop monitor service [ 3.136834] Initializing XFRM netlink socket [ 3.137249] NET: Registered protocol family 10 [ 3.140589] Segment Routing with IPv6 [ 3.152568] NET: Registered protocol family 17 [ 3.155656] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.156536] mpls_gso: MPLS GSO support [ 3.162938] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.170256] RAS: Correctable Errors collector initialized. [ 3.173177] AVX version of gcm_enc/dec engaged. [ 3.175033] AES CTR mode by8 optimization enabled [ 3.278426] sched_clock: Marking stable (3278347166, 0)->(4237020852, -958673686) [ 3.284255] registered taskstats version 1 [ 3.288058] Loading compiled-in X.509 certificates [ 3.289793] zswap: loaded using pool lzo/zbud [ 3.326641] Key type big_key registered [ 3.338715] Key type encrypted registered [ 3.341187] ima: No TPM chip found, activating TPM-bypass! [ 3.343564] ima: Allocated hash algorithm: sha1 [ 3.345752] ima: No architecture policies found [ 3.347760] evm: Initialising EVM extended attributes: [ 3.349906] evm: security.selinux [ 3.351231] evm: security.ima [ 3.352313] evm: security.capability [ 3.353401] evm: HMAC attrs: 0x1 [ 3.355444] rtc_cmos 00:05: setting system clock to 2026-09-07 02:28:11 UTC (1788748091) [ 3.362649] debug: unmapping init [mem 0xffffffff8ba03000-0xffffffff8bbfffff] [ 3.365903] debug: unmapping init [mem 0xffffffff8a782000-0xffffffff8aa58fff] [ 3.375069] Write protecting the kernel read-only data: 28672k [ 3.378472] debug: unmapping init [mem 0xffffffff88e03000-0xffffffff88ffffff] [ 3.381601] debug: unmapping init [mem 0xffffffff89714000-0xffffffff897fffff] [ 3.411430] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.418665] systemd[1]: Detected virtualization kvm. [ 3.420365] systemd[1]: Detected architecture x86-64. [ 3.421804] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.446803] systemd[1]: No hostname configured. [ 3.448211] systemd[1]: Set hostname to . [ 3.449937] random: systemd: uninitialized urandom read (16 bytes read) [ 3.452392] systemd[1]: Initializing machine ID from random generator. [ 3.502306] random: ln: uninitialized urandom read (6 bytes read) [ 3.598814] random: systemd: uninitialized urandom read (16 bytes read) [ 3.601611] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.611272] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.618977] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.441560] device-mapper: uevent: version 1.0.3 [ 4.444168] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 4.516965] hrtimer: interrupt took 2994395 ns [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.908975] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.350160] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 5.509975] scsi host0: ata_piix [ 5.525234] scsi host1: ata_piix [ 5.527191] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.531068] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.404460] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.862345] random: crng init done [ 9.863935] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.757435] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.296586] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.578038] SELinux: Disabled at runtime. [ 12.638197] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.650301] systemd[1]: Detected virtualization kvm. [ 12.652109] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.204259] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.207954] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.215679] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.220451] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.224985] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.235739] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.246286] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 13.363155] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started Flush Journal to Persistent Storage. [ 14.069761] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Mounted /mnt. [ OK ] Started udev Kernel Device Manager. [ 14.492386] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.514343] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.953226] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.964600] EDAC sbridge: Ver: 1.1.2 [ 16.797780] Key type dns_resolver registered [ 17.158194] NFS: Registering the id_resolver key type [ 17.160170] Key type id_resolver registered [ 17.161673] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg324-client login: [ 51.606531] libcfs: loading out-of-tree module taints kernel. [ 51.738658] Key type ._llcrypt registered [ 51.740033] Key type .llcrypt registered [ 52.174538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 52.182951] alg: No test for adler32 (adler32-zlib) [ 53.536439] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 54.428963] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 56.177761] Key type lgssc registered [ 58.509140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 187.642258] Lustre: Mounted lustre-client - version 2.17.58_41_g2290de3 [ 192.490549] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 211.224620] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing check_logdir /tmp/testlogs/ [ 213.476035] Lustre: lustre-OST0000-osc-ffff9eab47e4d000: disconnect after 23s idle [ 215.432578] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing yml_node [ 219.540425] Lustre: DEBUG MARKER: Client: 2.17.58.41 [ 221.842443] Lustre: DEBUG MARKER: MDS: 2.17.58.41 [ 224.249599] Lustre: DEBUG MARKER: OSS: 2.17.58.41 [ 225.591825] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Sep 6 22:31:52 EDT 2026 [ 241.965192] Lustre: Unmounted lustre-client [ 258.153497] Key type lgssc unregistered [ 258.372741] LNet: 8440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.384255] LNetError: 8440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 258.410519] LNet: Removed LNI 192.168.203.24@tcp [ 259.059447] Key type .llcrypt unregistered [ 259.061994] Key type ._llcrypt unregistered [ 280.790200] Key type ._llcrypt registered [ 280.791685] Key type .llcrypt registered [ 280.964939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 280.975973] alg: No test for adler32 (adler32-zlib) [ 281.812339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 281.863790] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 281.884471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 285.029567] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 285.237602] LNet: 9318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.241927] LNetError: 9318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 285.258369] LNet: Removed LNI 192.168.203.24@tcp [ 285.865385] Key type .llcrypt unregistered [ 285.869048] Key type ._llcrypt unregistered [ 291.704663] Key type ._llcrypt registered [ 291.707385] Key type .llcrypt registered [ 293.006388] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 22:32:59 (1788748379) [ 293.366152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 293.381573] alg: No test for adler32 (adler32-zlib) [ 294.275126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.391525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.411225] LNet: 9726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.428572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.498300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 294.516721] LNet: 9733:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 300.634404] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 22:33:07 (1788748387) [ 300.894921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.914174] LNet: 10310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.939765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.015747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 308.906233] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 22:33:15 (1788748395) [ 309.308513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.356993] LNet: 10896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.394633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.507592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 317.346158] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 22:33:24 (1788748404) [ 317.591408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.620091] LNet: 11481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.654656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.799369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 325.418724] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 22:33:31 (1788748411) [ 325.802588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.824451] LNet: 12066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.848583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.943849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 326.205335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 332.624694] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 22:33:39 (1788748419) [ 332.777909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.837660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.960850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 333.474672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 333.801964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 340.519909] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 22:33:47 (1788748427) [ 340.671764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.701901] LNet: 13250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.707733] LNet: 13250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 340.754176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.858573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 347.404037] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 22:33:54 (1788748434) [ 347.725623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.794854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.893754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 347.967111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 348.489838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 348.539709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 348.996739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 349.041195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 349.478992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 349.531610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 350.068299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 350.117419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 350.659593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 350.711293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 351.123827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 351.167804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 351.581154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 351.617808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 352.074872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 352.115369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 352.652679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 352.771175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 353.330924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 353.367845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 353.893349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 353.930941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 354.416182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 354.485469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 360.914433] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 22:34:07 (1788748447) [ 361.059683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.105245] LNet: 14517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.112164] LNet: 14517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 361.146941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.264433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 361.311877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 368.046525] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 22:34:14 (1788748454) [ 368.148145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.207929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.301788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 368.361306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 374.939344] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 22:34:21 (1788748461) [ 375.047670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.111120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.200941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 375.250489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 381.841353] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 22:34:28 (1788748468) [ 381.987332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.103312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.266799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 382.348143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 389.547841] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 22:34:36 (1788748476) [ 389.658211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.728520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.816199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 389.894051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 395.629610] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 22:34:42 (1788748482) [ 395.934641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.980676] LNet: 17453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.985243] LNet: 17453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 396.004639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.064682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 396.109110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 403.085118] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 22:34:49 (1788748489) [ 403.210446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.268377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.376750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 403.834027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 411.249194] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 22:34:57 (1788748497) [ 411.407201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.506488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.824420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 412.268513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.865374] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 22:35:06 (1788748506) [ 419.985359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.051590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.226971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 420.487629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.353563] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 22:35:14 (1788748514) [ 427.826115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.865477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.325529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.609434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 435.143595] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 22:35:22 (1788748522) [ 435.252695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.292474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.362228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.411268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.677310] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 22:35:28 (1788748528) [ 442.013543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.061802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.129795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.179182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 449.237779] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 22:35:36 (1788748536) [ 449.448731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.501790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.593270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 455.997768] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 22:35:42 (1788748542) [ 456.117462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.154224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.244837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 462.287350] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 22:35:49 (1788748549) [ 462.356384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.391295] LNet: 22881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 462.398378] LNet: 22881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 462.416928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.498683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 462.929117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 468.685551] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 22:35:55 (1788748555) [ 468.804245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.846835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.901954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 468.929441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 474.932574] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 22:36:01 (1788748561) [ 475.027604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.069843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.151577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 475.178823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 481.008956] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 22:36:08 (1788748568) [ 481.226877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.262138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.327906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 481.369050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 487.655561] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 22:36:14 (1788748574) [ 487.723639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.758498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.840508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 487.879320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 494.345951] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 22:36:21 (1788748581) [ 494.532798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.573094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.614335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 494.680711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 494.701597] LNetError: 25838:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 494.740488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 494.768260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 500.187162] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 22:36:27 (1788748587) [ 500.446638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.517761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.636816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mR55cUYkLj [ 500.756348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mR55cUYkLj [ 500.845258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mR55cUYkLj [ 506.128374] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 22:36:33 (1788748593) [ 506.220387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.267658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.336437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 506.377133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 506.450791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 512.118747] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 22:36:38 (1788748598) [ 512.221066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.277812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.391353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 512.415268] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 513.722031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 514.870369] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 516.848289] Key type lgssc registered [ 517.932368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 520.872970] LNetError: 27598:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.24@tcp99 has no available nets [ 523.117659] Key type lgssc unregistered [ 523.476505] LNetError: 28704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 523.494211] LNet: Removed LNI 192.168.203.24@tcp [ 523.903206] Key type .llcrypt unregistered [ 523.905745] Key type ._llcrypt unregistered [ 529.539924] Key type ._llcrypt registered [ 529.542399] Key type .llcrypt registered [ 530.670319] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 22:36:57 (1788748617) [ 531.203634] Key type .llcrypt unregistered [ 531.211552] Key type ._llcrypt unregistered [ 537.012387] Key type ._llcrypt registered [ 537.017622] Key type .llcrypt registered [ 537.191532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 537.216867] alg: No test for adler32 (adler32-zlib) [ 538.186270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.297045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 538.324692] LNet: 29614:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 538.335372] LNet: 29614:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 538.344653] LNet: 29614:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 538.367504] LNet: Message forwarding will be enabled in 2s [ 541.472161] LNet: Message forwarding enabled [ 541.912494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 541.945698] LNet: Message forwarding will stop in 2s to 6s [ 545.571453] LNet: Message forwarding disabled [ 546.602676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 546.628325] LNet: 29712:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 546.635119] LNet: 29712:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 546.644708] LNet: 29712:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 546.663974] LNet: Message forwarding will be enabled in 2s [ 546.844604] LNet: 29725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.436183] Key type .llcrypt unregistered [ 547.439145] Key type ._llcrypt unregistered [ 557.573474] Key type ._llcrypt registered [ 557.577479] Key type .llcrypt registered [ 558.784921] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 22:37:25 (1788748645) [ 559.077533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 559.089460] alg: No test for adler32 (adler32-zlib) [ 559.975589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.040609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 565.972757] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 22:37:32 (1788748652) [ 566.171194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.206563] LNet: 31141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.239214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.292909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 566.323424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 566.361839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 566.398367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 566.429905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 566.453295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 566.477654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 566.505304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 566.536947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 566.571362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 566.618644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 566.647614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 566.677443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 566.703695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 566.734363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 566.770836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 566.820955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 566.873877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 566.912685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 566.935674] LNetError: 31184:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 572.564911] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 22:37:39 (1788748659) [ 572.779241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.821516] LNet: 31754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.846738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.909883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 579.391840] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 22:37:46 (1788748666) [ 581.050402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg324-server [ 582.988528] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 22:37:49 (1788748669) [ 583.127213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.160371] LNet: 32683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.184236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.263255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 583.322766] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 584.215251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[25-29]@tcp [ 584.245869] LNet: 32818:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 584.298555] LNetError: 32689:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.28@tcp is being used as a gateway but routing feature is not turned on [ 584.317095] LNetError: 32689:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.28@tcp has gone from up to down [ 589.496757] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 22:37:56 (1788748676) [ 590.977159] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg324-server [ 592.540470] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 22:37:59 (1788748679) [ 592.794400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.828637] LNet: 33754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.836779] LNetError: 33754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.863772] LNet: Removed LNI 192.168.203.24@tcp [ 592.882765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.947758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.980056] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 593.493778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[25-30/2]@tcp [ 593.513772] LNet: 33871:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 593.520520] LNet: 33871:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 593.533141] LNetError: 33758:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 593.548800] LNetError: 33758:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 593.555795] LNetError: 33758:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.29@tcp has gone from up to down [ 593.556422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[25-30/2]@tcp [ 593.579246] LNetError: 33758:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 599.523426] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 22:38:06 (1788748686) [ 599.704085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.739420] LNet: 34463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.761818] LNetError: 34463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 599.798913] LNet: Removed LNI 192.168.203.24@tcp [ 599.834016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.979214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 600.059538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.103975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.152706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 600.207578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.252849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.314333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 600.381159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.411304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.463712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 600.549456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.604522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.654858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 600.751697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.793839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.845444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 600.938519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.983386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.036075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 601.108819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.137264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.200854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 601.320753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.371781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.436673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 601.544346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.593293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.673099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 601.777378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.814929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.891699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 601.945741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.989594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.058223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 608.433786] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 22:38:14 (1788748694) [ 610.566980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg324-server [ 612.193991] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 22:38:19 (1788748699) [ 613.700779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg324-server [ 615.411871] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 22:38:22 (1788748702) [ 615.487271] LNet: 35855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.493434] LNet: 35855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 616.152936] Key type .llcrypt unregistered [ 616.158158] Key type ._llcrypt unregistered [ 621.864209] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 622.031951] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 622.046182] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 622.054034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 624.634760] Key type ._llcrypt registered [ 624.639922] Key type .llcrypt registered [ 624.824964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 624.853842] alg: No test for adler32 (adler32-zlib) [ 625.818226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.891125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 625.916942] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 626.794746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 626.820464] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 628.781875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 628.800889] LNet: Removed LNI 10.1.2.3@tcp [ 628.925941] LNet: 36581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.931773] LNetError: 36581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.947675] LNet: Removed LNI 192.168.203.24@tcp [ 629.476873] Key type .llcrypt unregistered [ 629.478677] Key type ._llcrypt unregistered [ 639.211257] Key type ._llcrypt registered [ 639.213860] Key type .llcrypt registered [ 640.525456] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 22:38:47 (1788748727) [ 640.955369] Key type .llcrypt unregistered [ 640.960264] Key type ._llcrypt unregistered [ 646.760135] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 647.006214] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 647.014768] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 647.019310] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 649.497356] Key type ._llcrypt registered [ 649.500667] Key type .llcrypt registered [ 649.656873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 649.674341] alg: No test for adler32 (adler32-zlib) [ 650.575295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.650795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 650.669413] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 651.323233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 651.351096] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 653.698406] LNet: 38067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.716767] LNetError: 38067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 653.723673] LNet: Removed LNI 10.1.2.3@tcp [ 653.742701] LNet: Removed LNI 192.168.203.24@tcp [ 654.311993] Key type .llcrypt unregistered [ 654.314411] Key type ._llcrypt unregistered [ 665.423745] Key type ._llcrypt registered [ 665.426484] Key type .llcrypt registered [ 666.492811] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 22:39:13 (1788748753) [ 667.218756] Key type .llcrypt unregistered [ 667.229463] Key type ._llcrypt unregistered [ 673.026109] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 673.209404] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 673.217947] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 673.226168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 675.772599] Key type ._llcrypt registered [ 675.775577] Key type .llcrypt registered [ 676.009372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 676.021975] alg: No test for adler32 (adler32-zlib) [ 676.974413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.024596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 677.054030] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 678.659952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 678.696413] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 680.308231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 680.325708] LNet: Removed LNI 10.1.2.3@tcp [ 680.337813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 680.361226] LNet: Removed LNI 10.1.2.31@tcp [ 680.373674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 680.397724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 681.091281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 681.121392] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 681.927652] LNet: 39981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.934852] LNetError: 39981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 681.940550] LNet: Removed LNI 10.1.2.31@tcp [ 681.955896] LNet: Removed LNI 10.1.2.3@tcp [ 682.475164] Key type .llcrypt unregistered [ 682.478620] Key type ._llcrypt unregistered [ 694.686647] Key type ._llcrypt registered [ 694.689883] Key type .llcrypt registered [ 696.011874] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 22:39:42 (1788748782) [ 696.502886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 696.527262] alg: No test for adler32 (adler32-zlib) [ 697.480857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.572977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.600592] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 698.600206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.619331] LNet: 40934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.624917] LNetError: 40934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.646409] LNet: Removed LNI 192.168.203.24@tcp [ 698.662450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.707339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.24@tcp [ 698.724731] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 699.489680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.24@tcp [ 699.543260] LNet: Removed LNI 192.168.203.24@tcp [ 699.638449] LNet: 41069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.360428] Key type .llcrypt unregistered [ 700.362730] Key type ._llcrypt unregistered [ 709.999337] Key type ._llcrypt registered [ 710.005935] Key type .llcrypt registered [ 711.503837] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 22:39:57 (1788748797) [ 713.330918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg324-server [ 715.228634] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 22:40:01 (1788748801) [ 715.844580] Key type .llcrypt unregistered [ 715.846538] Key type ._llcrypt unregistered [ 721.528928] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 721.711279] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 721.716445] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 721.721407] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 724.245691] Key type ._llcrypt registered [ 724.248248] Key type .llcrypt registered [ 724.415671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.434230] alg: No test for adler32 (adler32-zlib) [ 725.302594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.343726] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 725.871548] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 727.505282] LNet: 42902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.514577] LNetError: 42902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.519836] LNet: Removed LNI 192.168.203.24@tcp1 [ 727.538811] LNet: Removed LNI 10.1.2.3@tcp2 [ 728.068700] Key type .llcrypt unregistered [ 728.071873] Key type ._llcrypt unregistered [ 737.168619] Key type ._llcrypt registered [ 737.170499] Key type .llcrypt registered [ 738.348386] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 22:40:25 (1788748825) [ 738.820806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 738.831587] alg: No test for adler32 (adler32-zlib) [ 739.732993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.772586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.806418] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 740.481501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.506839] LNet: 43871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.513274] LNetError: 43871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.532176] LNet: Removed LNI 192.168.203.24@tcp [ 740.545969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.614768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 740.646778] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 746.358067] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 22:40:33 (1788748833) [ 746.558140] LNet: 44460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.596160] LNetError: 44460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.616823] LNet: Removed LNI 192.168.203.24@tcp [ 747.393776] Key type .llcrypt unregistered [ 747.399839] Key type ._llcrypt unregistered [ 753.004397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 753.229579] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 753.236555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 753.245047] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 755.584702] Key type ._llcrypt registered [ 755.588707] Key type .llcrypt registered [ 755.725172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.735593] alg: No test for adler32 (adler32-zlib) [ 756.639339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.705760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.730883] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 757.406311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 757.426558] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.516623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.535921] LNet: 45199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.542236] LNetError: 45199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 759.549177] LNet: Removed LNI 10.1.2.3@tcp [ 759.565732] LNet: Removed LNI 192.168.203.24@tcp [ 759.597987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.667114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 759.698651] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 759.705370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.104923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.126564] LNet: 45226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.132677] LNetError: 45226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 760.139348] LNetError: 45226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 760.144758] LNet: Removed LNI 10.1.2.3@tcp [ 760.154284] LNet: Removed LNI 192.168.203.24@tcp [ 760.176449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.253978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 760.279157] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 760.282292] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.635740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 760.652745] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 761.530192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.556891] LNet: 45388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.565372] LNetError: 45388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 761.571736] LNetError: 45388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 761.578340] LNet: Removed LNI 10.1.2.3@tcp [ 761.580361] LNet: Removed LNI 192.168.203.24@tcp [ 761.593298] LNet: Removed LNI 192.168.203.24@tcp2 [ 761.610244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.671300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 761.691311] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 761.696117] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.700890] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 766.904650] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 22:40:53 (1788748853) [ 767.064644] LNet: 45985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.069375] LNetError: 45985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 767.074128] LNetError: 45985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 767.090275] LNet: Removed LNI 10.1.2.3@tcp [ 767.093700] LNet: Removed LNI 192.168.203.24@tcp [ 767.109686] LNet: Removed LNI 192.168.203.24@tcp2 [ 767.648909] Key type .llcrypt unregistered [ 767.653978] Key type ._llcrypt unregistered [ 772.927575] Key type ._llcrypt registered [ 772.930199] Key type .llcrypt registered [ 773.111600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 773.133927] alg: No test for adler32 (adler32-zlib) [ 774.018484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.070530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 774.105317] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 774.759168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.787805] LNet: 46377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.795864] LNetError: 46377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.830580] LNet: Removed LNI 192.168.203.24@tcp [ 774.883574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.985781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 775.023903] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 775.530141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.551925] LNet: 46407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.561315] LNetError: 46407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.575935] LNet: Removed LNI 192.168.203.24@tcp [ 775.599291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.691166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 775.719078] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 776.156812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.193036] LNet: Removed LNI 192.168.203.24@tcp [ 776.217960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.270185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 776.303553] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 776.722377] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 776.803077] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 776.806871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 776.812032] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 779.146870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.166874] LNet: 46515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.174178] LNet: 46515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 779.179672] LNetError: 46515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.184323] LNetError: 46515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 779.195372] LNet: Removed LNI 192.168.203.24@tcp [ 779.221907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.297827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 779.319334] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 785.426905] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 22:41:12 (1788748872) [ 785.491808] LNet: 47105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.500990] LNetError: 47105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.511154] LNet: Removed LNI 192.168.203.24@tcp [ 785.997398] Key type .llcrypt unregistered [ 785.999241] Key type ._llcrypt unregistered [ 791.046586] Key type ._llcrypt registered [ 791.052523] Key type .llcrypt registered [ 791.259538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 791.274444] alg: No test for adler32 (adler32-zlib) [ 792.181841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.246116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 792.276476] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 793.138433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 793.178972] LNet: 47509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.184725] LNetError: 47509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.200867] LNet: Removed LNI 192.168.203.24@tcp [ 793.225702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.340095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 793.367104] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 798.731944] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 22:41:25 (1788748885) [ 799.113684] LNet: 48101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.130465] LNetError: 48101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.146475] LNet: Removed LNI 192.168.203.24@tcp [ 799.925877] Key type .llcrypt unregistered [ 799.928967] Key type ._llcrypt unregistered [ 805.271751] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 805.406334] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 805.422117] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 805.427743] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 808.005894] Key type ._llcrypt registered [ 808.009427] Key type .llcrypt registered [ 808.244595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 808.276330] alg: No test for adler32 (adler32-zlib) [ 809.221801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.272670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.303622] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 810.131695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.148410] LNet: 48548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.159323] LNetError: 48548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.186716] LNet: Removed LNI 192.168.203.24@tcp [ 810.241863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.324745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 810.348802] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 815.906305] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 22:41:42 (1788748902) [ 816.046263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.087703] LNet: 49147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.094865] LNetError: 49147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.104945] LNet: Removed LNI 192.168.203.24@tcp [ 816.125824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.204621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 821.183612] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 22:41:48 (1788748908) [ 821.286093] LNet: 49725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.799706] Key type .llcrypt unregistered [ 821.803037] Key type ._llcrypt unregistered [ 826.309923] Key type ._llcrypt registered [ 826.314214] Key type .llcrypt registered [ 826.478596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 826.495525] alg: No test for adler32 (adler32-zlib) [ 827.383364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.427224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 827.449173] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 828.117937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 828.138534] LNet: 50124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.143634] LNetError: 50124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.153580] LNet: Removed LNI 192.168.203.24@tcp [ 828.163886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.200153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 828.221616] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 833.116925] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 22:42:00 (1788748920) [ 833.180229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.200851] LNet: 50710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.206618] LNetError: 50710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.220890] LNet: Removed LNI 192.168.203.24@tcp [ 833.251454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.349156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 837.850144] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 22:42:04 (1788748924) [ 838.810817] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 840.191582] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 22:42:07 (1788748927) [ 841.217398] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 842.512383] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 22:42:09 (1788748929) [ 843.532921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 844.929199] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 22:42:11 (1788748931) [ 845.998710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 847.406523] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 22:42:14 (1788748934) [ 848.735844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 850.263496] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 22:42:17 (1788748937) [ 850.377715] LNet: 52736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.865472] Key type .llcrypt unregistered [ 850.869524] Key type ._llcrypt unregistered [ 855.221112] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.395929] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.402333] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.411451] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 857.774173] Key type ._llcrypt registered [ 857.775903] Key type .llcrypt registered [ 857.891152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 857.908591] alg: No test for adler32 (adler32-zlib) [ 858.761747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.831559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 858.863800] LNet: Added LNI 192.168.203.24@tcp [8/512/0/180] [ 858.872321] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 858.875954] LNet: 53057:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 858.949653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.965984] LNet: 53076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.972370] LNetError: 53076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 858.980571] LNet: Removed LNI 10.1.2.3@tcp [ 858.982304] LNetError: 53076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.994637] LNet: Removed LNI 192.168.203.24@tcp [ 859.008648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.056966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 859.076892] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 859.119815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.136983] LNet: 53100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.142829] LNetError: 53100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.160951] LNet: Removed LNI 192.168.203.24@tcp [ 859.175733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.234618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 859.251716] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 859.258722] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 859.261667] LNet: 53108:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 859.329085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.343458] LNet: 53126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.349754] LNetError: 53126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 859.354862] LNet: Removed LNI 10.1.2.3@tcp [ 859.356867] LNetError: 53126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.368996] LNet: Removed LNI 192.168.203.24@tcp [ 859.389150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.426896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 859.448655] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 859.451908] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 859.456116] LNet: 53134:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 859.583373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.599972] LNet: 53160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.609319] LNetError: 53160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 859.617552] LNet: Removed LNI 10.1.2.3@tcp [ 859.619381] LNetError: 53160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.637076] LNet: Removed LNI 192.168.203.24@tcp [ 859.654762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.716360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 859.753936] LNet: Added LNI 192.168.203.24@tcp [8/256/0/182] [ 859.758572] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 859.833412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.850562] LNet: 53188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.856152] LNetError: 53188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 859.861342] LNet: Removed LNI 10.1.2.3@tcp [ 859.863749] LNetError: 53188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.892108] LNet: Removed LNI 192.168.203.24@tcp [ 859.923253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.970614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 859.986483] LNet: Added LNI 192.168.203.24@tcp [8/256/0/182] [ 859.993286] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 859.995166] LNet: 53196:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 860.063972] LNet: Removed LNI 10.1.2.3@tcp [ 860.074282] LNet: Removed LNI 192.168.203.24@tcp [ 860.516295] Key type .llcrypt unregistered [ 860.520674] Key type ._llcrypt unregistered [ 868.661675] Key type ._llcrypt registered [ 868.663880] Key type .llcrypt registered [ 869.542972] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 22:42:36 (1788748956) [ 870.587843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 871.776725] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 22:42:38 (1788748958) [ 872.106934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 872.120051] alg: No test for adler32 (adler32-zlib) [ 873.009295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.044378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.059682] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 873.577736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.24 [ 873.593722] LNet: 54457:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 873.603448] LNet: Removed LNI 192.168.203.24@tcp [ 877.658678] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 22:42:44 (1788748964) [ 877.777126] LNet: 55022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.789798] LNetError: 55022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.809117] LNet: Removed LNI 192.168.203.24@tcp [ 878.336843] Key type .llcrypt unregistered [ 878.341636] Key type ._llcrypt unregistered [ 882.787342] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 882.995718] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 885.499662] Key type ._llcrypt registered [ 885.501540] Key type .llcrypt registered [ 885.639543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 885.651811] alg: No test for adler32 (adler32-zlib) [ 886.885545] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 891.644238] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 22:42:58 (1788748978) [ 891.768869] LNet: 55932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.779831] LNetError: 55932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 891.808174] LNet: Removed LNI 10.1.2.3@tcp [ 892.317256] Key type .llcrypt unregistered [ 892.320380] Key type ._llcrypt unregistered [ 896.435541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 896.658389] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 896.660992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 896.666554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 899.153869] Key type ._llcrypt registered [ 899.156120] Key type .llcrypt registered [ 899.272768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 899.284460] alg: No test for adler32 (adler32-zlib) [ 900.172241] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 904.638461] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 22:43:11 (1788748991) [ 905.665593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 906.751159] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 22:43:14 (1788748994) [ 906.797217] LNet: 57126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.802769] LNetError: 57126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 906.824472] LNet: Removed LNI 10.1.2.3@tcp [ 907.181719] Key type .llcrypt unregistered [ 907.184250] Key type ._llcrypt unregistered [ 910.431337] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.643934] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.648702] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.652829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 913.124206] Key type ._llcrypt registered [ 913.126089] Key type .llcrypt registered [ 913.254169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.264302] alg: No test for adler32 (adler32-zlib) [ 914.132567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.176672] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.850506] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 22:43:27 (1788749007) [ 920.219066] LNet: 58359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.226670] LNetError: 58359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.248281] LNet: Removed LNI 10.1.2.3@tcp [ 920.652894] Key type .llcrypt unregistered [ 920.655754] Key type ._llcrypt unregistered [ 924.775235] Key type ._llcrypt registered [ 924.777336] Key type .llcrypt registered [ 924.934673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 924.947704] alg: No test for adler32 (adler32-zlib) [ 925.843204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 925.870382] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 931.512897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 932.129631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_interrupt -r 1 [ 932.170434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_interrupt -r 1 [ 932.211867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 932.240465] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 932.402191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 932.429291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 932.462055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 932.496171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 932.525615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 932.547648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 932.565971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 932.589073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 932.617682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 932.636150] LNet: 58795:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 932.655943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 932.696037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 932.727985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 932.768671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 932.809991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 932.842630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 932.863908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 933.295830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_dropped -r 1 [ 933.331281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_dropped -r 1 [ 933.360937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 933.390774] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 933.561781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 933.581129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 933.606904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 933.636204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 933.671306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 933.705902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 933.733199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 933.765083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 933.793129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 933.813734] LNet: 58907:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 933.835536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 933.871244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 933.901181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 933.926261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 933.952519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 933.988657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 934.028808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 934.539455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_aborted -r 1 [ 934.570870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_aborted -r 1 [ 934.622654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 934.641817] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 934.824150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 934.852431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 934.879261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 934.900752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 934.929251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 934.960719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 934.987448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 935.014313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 935.043617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 935.055788] LNet: 59018:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 935.076384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 935.103688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 935.132330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 935.140617] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 935.164717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 935.192751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 935.223237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 935.245714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 935.619954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_no_route -r 1 [ 935.655290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_no_route -r 1 [ 935.679965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 935.842061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 935.864852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 935.895493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 935.918528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 935.943410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 935.967570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 935.989778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 936.018133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 936.045133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 936.075539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 936.097986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 936.121865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 936.150914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 936.187271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 936.214793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 936.240267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 936.577487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_timeout -r 1 [ 936.608471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_timeout -r 1 [ 936.641451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 936.665407] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 936.671653] LNet: Skipped 2 previous similar messages [ 936.831278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 936.858528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 936.887306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 936.919396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 936.950441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 936.975418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 937.005462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 937.032262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 937.061988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 937.081579] LNet: 59240:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 937.089448] LNet: 59240:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 937.102860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 937.119804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 937.139924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 937.161993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 937.184937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 937.217788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 937.244565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 937.610819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_error -r 1 [ 937.647591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_error -r 1 [ 937.697512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 937.925756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 937.957272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 937.983841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 938.005695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 938.029553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 938.053831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 938.079558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 938.110313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 938.139786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 938.172823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 938.215280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 938.243378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 938.268704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 938.297955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 938.329312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 938.355808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 938.525732] LNet: 59393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.530912] LNetError: 59393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.545784] LNet: Removed LNI 192.168.203.24@tcp [ 938.797500] Key type .llcrypt unregistered [ 938.799723] Key type ._llcrypt unregistered [ 949.631918] Key type ._llcrypt registered [ 949.633965] Key type .llcrypt registered [ 950.600129] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 22:43:57 (1788749037) [ 951.275579] Key type .llcrypt unregistered [ 951.278370] Key type ._llcrypt unregistered [ 955.014988] Key type ._llcrypt registered [ 955.017832] Key type .llcrypt registered [ 955.157249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 955.165722] alg: No test for adler32 (adler32-zlib) [ 956.030730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 956.056925] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 961.831378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 969.700151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.736318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 969.757855] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 970.775343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_interrupt -r 1 [ 970.810049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 1 [ 970.843051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_interrupt -r 1 [ 970.882909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 1 [ 970.931599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_interrupt -r 1 [ 970.967509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 1 [ 970.988634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_interrupt -r 1 [ 971.026730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 1 [ 971.054597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 971.077705] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 971.082827] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 971.239206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 971.269921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 971.293837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 971.320099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 971.343978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 971.361798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 971.381914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 971.410943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 971.443847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 971.465655] LNet: 60855:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 971.486646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 971.504504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 971.525544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 971.558299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 971.601219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 971.632330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 971.658952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 972.046922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_dropped -r 1 [ 972.073537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 1 [ 972.104349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_dropped -r 1 [ 972.134510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 1 [ 972.179149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_dropped -r 1 [ 972.219104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 1 [ 972.255940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_dropped -r 1 [ 972.301387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 1 [ 972.341339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 972.361287] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 972.369582] LNet: Skipped 2 previous similar messages [ 972.552326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 972.577755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 972.606364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 972.645242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 972.680748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 972.710756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 972.743705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 972.773330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 972.803595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 972.825230] LNet: 60978:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 972.846932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 972.878610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 972.906879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 972.932810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 972.968354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 973.004324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 973.035268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 973.429501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_aborted -r 1 [ 973.456118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 1 [ 973.480469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_aborted -r 1 [ 973.506275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 1 [ 973.541591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_aborted -r 1 [ 973.587705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 1 [ 973.623635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_aborted -r 1 [ 973.652743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 1 [ 973.697284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 973.720345] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 973.726334] LNet: Skipped 2 previous similar messages [ 973.890128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 973.917809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 973.948565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 973.974717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 974.002232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 974.033400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 974.074542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 974.105688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 974.131398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 974.149381] LNet: 61101:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 974.165979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 974.194686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 974.231759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 974.271048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 974.304542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 974.325273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 974.348451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 974.754739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_no_route -r 1 [ 974.786107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 1 [ 974.829700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_no_route -r 1 [ 974.871823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 1 [ 974.908823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_no_route -r 1 [ 974.956860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 1 [ 974.986839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_no_route -r 1 [ 975.018863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 1 [ 975.054996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 975.237897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 975.281714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 975.318477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 975.343720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 975.369965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 975.397321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 975.436674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 975.468843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 975.504576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 975.526911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 975.560742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 975.590490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 975.613179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 975.642640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 975.679268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 975.713393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 976.064545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_timeout -r 1 [ 976.089946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 1 [ 976.116637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_timeout -r 1 [ 976.158259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 1 [ 976.194774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_timeout -r 1 [ 976.226105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 1 [ 976.246533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_timeout -r 1 [ 976.289928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 1 [ 976.326204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 976.347940] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 976.350919] LNet: Skipped 5 previous similar messages [ 976.534906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 976.563805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 976.590266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 976.615863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 976.635024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 976.656263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 976.682895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 976.706926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 976.732662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 976.745533] LNet: 61347:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 976.749469] LNet: 61347:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 976.762775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 976.787175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 976.808280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 976.833262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 976.857687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 976.882159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 976.903955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 977.289360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_error -r 1 [ 977.313939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_error -r 1 [ 977.346852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_error -r 1 [ 977.374570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_error -r 1 [ 977.401595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_error -r 1 [ 977.436934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_error -r 1 [ 977.466504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_error -r 1 [ 977.489888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_error -r 1 [ 977.525795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 977.679801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 977.698637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 977.725734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 977.758256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 977.783765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 977.807984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 977.829183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 977.865914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 977.895597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 977.924270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 977.951263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 977.992593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 978.019567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 978.047967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 978.098328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 978.131376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 981.831955] LNet: 61546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.846324] LNetError: 61546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.854574] LNet: Removed LNI 192.168.203.24@tcp [ 981.873205] LNet: Removed LNI 192.168.203.24@tcp1 [ 982.215971] Key type .llcrypt unregistered [ 982.217945] Key type ._llcrypt unregistered [ 993.406538] Key type ._llcrypt registered [ 993.408458] Key type .llcrypt registered [ 994.279836] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:44:41 (1788749081) [ 994.861620] Key type .llcrypt unregistered [ 994.869796] Key type ._llcrypt unregistered [ 999.619904] Key type ._llcrypt registered [ 999.623778] Key type .llcrypt registered [ 999.763279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 999.775178] alg: No test for adler32 (adler32-zlib) [ 1000.697135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1000.720162] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1006.120526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1006.573307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_dropped -r 1 [ 1006.603968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_dropped -r 1 [ 1006.641584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1006.661533] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1006.857500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1006.890915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1006.918275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1006.952549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1006.980932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1007.038691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1007.079262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1007.111725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1007.159244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1007.175406] LNet: 62790:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1007.196082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1007.229347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1007.259881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1007.287467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1007.312268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1007.333143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1007.363369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1007.727705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_error -r 1 [ 1007.757945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_error -r 1 [ 1007.806893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1007.826812] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1007.934485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1007.964947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1007.998298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1008.022665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1008.049653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1008.078545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1008.107521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1008.130605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1008.155299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1008.173681] LNet: 62901:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1008.188980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1008.212933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1008.242040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1008.269249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1008.290965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1008.314969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1008.340236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1008.680669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_timeout -r 1 [ 1008.704924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_timeout -r 1 [ 1008.732875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1008.871692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1008.893849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1008.918545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1008.938736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1008.964229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1008.988799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1009.016131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1009.047722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1009.079996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1009.110476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1009.150523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1009.174949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1009.205680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1009.235674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1009.283275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1009.318757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1009.485502] LNet: 63053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.490766] LNetError: 63053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.534694] LNet: Removed LNI 192.168.203.24@tcp [ 1010.848670] Key type .llcrypt unregistered [ 1010.852665] Key type ._llcrypt unregistered [ 1021.752204] Key type ._llcrypt registered [ 1021.754870] Key type .llcrypt registered [ 1022.659730] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:45:09 (1788749109) [ 1023.243934] Key type .llcrypt unregistered [ 1023.246282] Key type ._llcrypt unregistered [ 1027.440487] Key type ._llcrypt registered [ 1027.442905] Key type .llcrypt registered [ 1027.603672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.615580] alg: No test for adler32 (adler32-zlib) [ 1028.480815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1028.501673] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1033.672574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1039.732771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.781029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1039.814175] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1040.757937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_dropped -r 1 [ 1040.783959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 1 [ 1040.814563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_dropped -r 1 [ 1040.843570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 1 [ 1040.884527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_dropped -r 1 [ 1040.935067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 1 [ 1040.977974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_dropped -r 1 [ 1040.998965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 1 [ 1041.033640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1041.054048] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1041.063442] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.124@tcp [ 1041.254274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1041.284761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1041.313640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1041.348291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1041.375112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1041.415059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1041.437491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1041.459755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1041.481237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1041.497548] LNet: 64516:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1041.521912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1041.549703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1041.572548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1041.610721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1041.640989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1041.677150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1041.715266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1041.764192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1042.183368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_error -r 1 [ 1042.207248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_error -r 1 [ 1042.237820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_error -r 1 [ 1042.269642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_error -r 1 [ 1042.296643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_error -r 1 [ 1042.331579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_error -r 1 [ 1042.362748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_error -r 1 [ 1042.388411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_error -r 1 [ 1042.418511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1042.434821] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1042.439991] LNet: Skipped 2 previous similar messages [ 1042.648025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.683695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1042.715387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.735881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1042.755743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1042.776269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1042.801876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1042.833828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1042.860697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1042.883450] LNet: 64641:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1042.906839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1042.940674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1042.968592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.005719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.052842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.081210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.123729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1043.156613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.574385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_timeout -r 1 [ 1043.613201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_timeout -r 1 [ 1043.644633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_timeout -r 1 [ 1043.694664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_timeout -r 1 [ 1043.726428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_timeout -r 1 [ 1043.760251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_timeout -r 1 [ 1043.792766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_timeout -r 1 [ 1043.823718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_timeout -r 1 [ 1043.853240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1043.867350] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1044.031055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1044.052667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1044.076626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1044.094727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1044.118443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1044.144851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1044.163691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.188460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1044.217379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1044.243254] LNet: 64766:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1044.259404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1044.283444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.306358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.323560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.348924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.385594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.413451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1044.449696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.824827] LNet: 64844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.836255] LNetError: 64844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.847042] LNet: Removed LNI 192.168.203.24@tcp [ 1048.858482] LNet: Removed LNI 192.168.203.24@tcp1 [ 1049.215832] Key type .llcrypt unregistered [ 1049.218803] Key type ._llcrypt unregistered [ 1062.158672] Key type ._llcrypt registered [ 1062.162428] Key type .llcrypt registered [ 1063.245326] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:45:50 (1788749150) [ 1064.396836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1065.924892] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:45:52 (1788749152) [ 1066.527191] Key type .llcrypt unregistered [ 1066.530186] Key type ._llcrypt unregistered [ 1070.846200] Key type ._llcrypt registered [ 1070.848923] Key type .llcrypt registered [ 1071.007174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1071.021910] alg: No test for adler32 (adler32-zlib) [ 1071.872390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1071.898174] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1077.216040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1077.632316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e network_timeout -r 1 [ 1077.658726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e network_timeout -r 1 [ 1077.691846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1077.704866] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1077.838832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1077.865371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1077.892945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1077.917116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1077.941975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1077.964676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1077.982676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1078.007234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1078.032553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1078.048445] LNet: 66383:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1078.048603] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1078.074536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.099766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1078.137157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1078.161143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1078.178804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.199489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1078.369982] LNet: 66422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.375853] LNetError: 66422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.396770] LNet: Removed LNI 192.168.203.24@tcp [ 1078.653346] Key type .llcrypt unregistered [ 1078.656548] Key type ._llcrypt unregistered [ 1089.782918] Key type ._llcrypt registered [ 1089.786909] Key type .llcrypt registered [ 1089.904941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1089.914533] alg: No test for adler32 (adler32-zlib) [ 1090.772192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1090.796605] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1096.464569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1102.363091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1102.398725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1102.419777] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1103.390874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e network_timeout -r 1 [ 1103.413390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e network_timeout -r 1 [ 1103.443435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e network_timeout -r 1 [ 1103.478655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e network_timeout -r 1 [ 1103.508290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e network_timeout -r 1 [ 1103.544359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e network_timeout -r 1 [ 1103.576953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e network_timeout -r 1 [ 1103.619384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e network_timeout -r 1 [ 1103.672586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1103.696886] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1103.872686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.897722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.934092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.965626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.998291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1104.018764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.045674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.080040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.100288] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1104.107099] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.124@tcp [ 1104.113742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.127182] LNet: 67297:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.157386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.189868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.217426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.237915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.262450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.287161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.601388] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1107.609398] LNet: Skipped 2 previous similar messages [ 1107.794663] LNet: 67370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.800954] LNetError: 67370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.806993] LNet: Removed LNI 192.168.203.24@tcp [ 1107.818367] LNet: Removed LNI 192.168.203.24@tcp1 [ 1108.129782] Key type .llcrypt unregistered [ 1108.135736] Key type ._llcrypt unregistered [ 1119.118404] Key type ._llcrypt registered [ 1119.122029] Key type .llcrypt registered [ 1120.054951] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 22:46:47 (1788749207) [ 1120.380065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1120.390154] alg: No test for adler32 (adler32-zlib) [ 1121.236810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1121.271688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1121.289253] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1121.724519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1121.737491] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1122.277482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1122.335187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1122.374705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1122.406561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1122.432483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1122.454944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1122.478559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.24@tcp [ 1122.492725] LNetError: 68465:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.24@tcp to recovery [ 1123.297200] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1123.297420] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1125.344682] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1125.907971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1129.442120] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1130.352358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1130.373845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1130.423900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1130.448925] LNet: 68583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.456071] LNetError: 68583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1130.460974] LNet: Removed LNI 192.168.203.24@tcp [ 1130.473847] LNet: Removed LNI 192.168.203.24@tcp1 [ 1130.511240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1130.585830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1130.608897] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1131.092309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1131.107525] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1131.736184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1131.787618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.840821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1131.879841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1131.910938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1131.950706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1131.992303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1132.019333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.24@tcp [ 1132.035980] LNetError: 68856:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.24@tcp to recovery [ 1132.578082] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1135.425546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1138.722433] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1138.732388] LNet: Skipped 1 previous similar message [ 1143.059593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1143.094358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1143.139086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1148.107432] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 22:47:15 (1788749235) [ 1148.315763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1148.334446] LNet: 69573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.339612] LNetError: 69573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.345270] LNetError: 69573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1148.354218] LNet: Removed LNI 192.168.203.24@tcp [ 1148.375945] LNet: Removed LNI 192.168.203.24@tcp1 [ 1148.405525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1148.463984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1148.487090] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1149.048155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1149.065674] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1149.670957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1149.741996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1149.773706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1149.817943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1149.854717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1149.883843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1149.919855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.24@tcp --health 0 [ 1149.985213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1150.496301] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.24@tcp [ 1150.502395] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1150.519460] LNet: Skipped 1 previous similar message [ 1164.974691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1165.085631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1165.113821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.135528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.24@tcp --health 500 [ 1165.214441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1172.718733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1172.762820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1172.782982] LNet: 70072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.787530] LNetError: 70072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.790760] LNetError: 70072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1172.795406] LNet: Removed LNI 192.168.203.24@tcp [ 1172.808262] LNet: Removed LNI 192.168.203.24@tcp1 [ 1172.831926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1172.883306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1172.900675] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1173.382718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1173.399758] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1174.025894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1174.061610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.103493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1174.124327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1174.150802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1174.172479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1174.193650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1174.227491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.24@tcp --health 0 [ 1175.969323] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1175.977378] LNet: Skipped 3 previous similar messages [ 1176.564801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1186.370689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1186.398116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1186.422195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.753401] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 22:47:57 (1788749277) [ 1191.173666] LNet: 71128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.178305] LNetError: 71128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.182947] LNetError: 71128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1191.188942] LNet: Removed LNI 192.168.203.24@tcp [ 1191.200316] LNet: Removed LNI 192.168.203.24@tcp1 [ 1191.541115] Key type .llcrypt unregistered [ 1191.544312] Key type ._llcrypt unregistered [ 1195.213855] Key type ._llcrypt registered [ 1195.219044] Key type .llcrypt registered [ 1195.351393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1195.364975] alg: No test for adler32 (adler32-zlib) [ 1196.208210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1196.235813] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1201.975711] Lustre: DEBUG MARKER: Initial discovery [ 1202.022284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.124@tcp [ 1203.595543] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1203.774309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1203.798355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.124@tcp [ 1203.817551] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1203.839430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1203.865608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.124@tcp -r 1 -m GET -l 3 [ 1203.890983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.124@tcp -r 1 -m GET -e local_error [ 1203.917726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.124@tcp -r 1 -m PUT -l 6 [ 1204.455813] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1204.923871] Lustre: DEBUG MARKER: Force oleg324-client.virtnet to discover 192.168.203.124@tcp (in background) [ 1204.986993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.124@tcp [ 1206.037166] Lustre: DEBUG MARKER: Force oleg324-server to discover 192.168.203.24@tcp [ 1207.712339] Lustre: DEBUG MARKER: Wait for 71879 [ 1208.035468] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1208.986407] Lustre: DEBUG MARKER: Finished wait on 71879 [ 1209.050428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1209.078104] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1209.092109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1209.291642] LNet: 72336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.297913] LNetError: 72336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.311345] LNet: Removed LNI 192.168.203.24@tcp [ 1209.613460] Key type .llcrypt unregistered [ 1209.617938] Key type ._llcrypt unregistered [ 1220.791810] Key type ._llcrypt registered [ 1220.793492] Key type .llcrypt registered [ 1221.719540] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:48:28 (1788749308) [ 1222.024488] Key type .llcrypt unregistered [ 1222.027378] Key type ._llcrypt unregistered [ 1225.754217] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1225.851307] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1225.854817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1225.860505] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1228.293513] Key type ._llcrypt registered [ 1228.298466] Key type .llcrypt registered [ 1228.444631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1228.460594] alg: No test for adler32 (adler32-zlib) [ 1229.338025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1229.387871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1229.408806] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1229.964424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1229.979701] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1235.857354] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:48:43 (1788749323) [ 1235.972105] LNet: 74405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.982809] LNetError: 74405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1235.992216] LNet: Removed LNI 10.1.2.3@tcp [ 1236.009095] LNet: Removed LNI 192.168.203.24@tcp [ 1236.420695] Key type .llcrypt unregistered [ 1236.422278] Key type ._llcrypt unregistered [ 1241.302552] Key type ._llcrypt registered [ 1241.304798] Key type .llcrypt registered [ 1241.447976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1241.465100] alg: No test for adler32 (adler32-zlib) [ 1242.354619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.391791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1242.409934] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1242.892837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1242.917418] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1243.559393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.24@tcp [ 1244.640126] LNetError: Unexpected error -22 connecting to 192.168.203.24@tcp at host 192.168.203.24:988 [ 1247.858291] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:48:54 (1788749334) [ 1247.939649] LNet: 75526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.945208] LNetError: 75526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1247.952605] LNet: Removed LNI 10.1.2.3@tcp [ 1247.964900] LNet: Removed LNI 192.168.203.24@tcp [ 1248.395464] Key type .llcrypt unregistered [ 1248.402173] Key type ._llcrypt unregistered [ 1252.316326] Key type ._llcrypt registered [ 1252.320142] Key type .llcrypt registered [ 1252.519187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1252.533909] alg: No test for adler32 (adler32-zlib) [ 1253.415489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1253.466426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1253.491934] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1253.961259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1253.973541] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1254.495356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.24@tcp1 --nid 192.168.203.24@tcp2 [ 1257.056762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp1 192.168.203.24@tcp2 [ 1257.071982] LNetError: 76365:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.24@tcp1 to 192.168.203.24@tcp2 - Route table may be misconfigured [ 1257.096425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp2 192.168.203.24@tcp1 [ 1261.229034] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:49:08 (1788749348) [ 1261.303992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1261.332983] LNet: 76932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.346266] LNetError: 76932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.351758] LNet: Removed LNI 192.168.203.24@tcp1 [ 1262.377422] LNet: Removed LNI 192.168.203.24@tcp2 [ 1262.406589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1262.452607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1262.469511] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1262.945938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1262.961043] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1263.539961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1263.575141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.24@tcp [ 1263.617411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp1 192.168.203.24@tcp1 [ 1263.657382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.24@tcp -d 192.168.203.24@tcp -e network_timeout [ 1263.699476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -e network_timeout [ 1263.728246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -e network_timeout [ 1263.757222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -e network_timeout [ 1263.780031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1263.794645] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1263.812996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1263.882043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1263.949648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1264.480698] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1268.372442] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:49:15 (1788749355) [ 1268.425798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1268.450684] LNet: 77793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.462822] LNetError: 77793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.474728] LNetError: 77793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1268.485849] LNet: Removed LNI 192.168.203.24@tcp [ 1269.543316] LNet: Removed LNI 192.168.203.24@tcp1 [ 1269.563314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.638819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1269.805739] LNet: 77836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.136786] Key type .llcrypt unregistered [ 1270.140655] Key type ._llcrypt unregistered [ 1277.220628] Key type ._llcrypt registered [ 1277.222315] Key type .llcrypt registered [ 1278.130653] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:49:25 (1788749365) [ 1278.387977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1278.411298] alg: No test for adler32 (adler32-zlib) [ 1279.258078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.366390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1281.932594] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:49:28 (1788749368) [ 1281.996123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1282.010518] LNet: 78922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.042619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1282.093664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1282.134906] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1282.781406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1282.796629] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1283.465875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1283.495776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp1 [ 1283.529131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp1 [ 1287.768408] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:49:35 (1788749375) [ 1289.103977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg324-server [ 1290.330737] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:49:37 (1788749377) [ 1291.515326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg324-server [ 1292.739939] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:49:39 (1788749379) [ 1294.037199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg324-server [ 1295.272089] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:49:42 (1788749382) [ 1296.710541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg324-server [ 1297.961124] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:49:45 (1788749385) [ 1299.165588] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg324-server [ 1300.521252] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:49:47 (1788749387) [ 1301.770573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg324-server [ 1303.089642] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:49:50 (1788749390) [ 1304.596185] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg324-server [ 1305.874520] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:49:53 (1788749393) [ 1307.356783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg324-server [ 1308.564372] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:49:55 (1788749395) [ 1309.950530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg324-server [ 1311.362514] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:49:58 (1788749398) [ 1311.457063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1311.484580] LNet: 82801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.495679] LNetError: 82801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.510983] LNet: Removed LNI 192.168.203.24@tcp [ 1312.550927] LNet: Removed LNI 192.168.203.24@tcp1 [ 1312.583466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.638472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1312.658066] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1313.306395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1313.375452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1313.588929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1313.618445] LNet: 82952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.630555] LNetError: 82952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.634149] LNetError: 82952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1313.645605] LNet: Removed LNI 192.168.203.24@tcp [ 1313.673708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1313.745099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1313.777410] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1314.664957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1314.858801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1314.884334] LNetError: 83098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.942783] LNet: Removed LNI 192.168.203.24@tcp [ 1315.961850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1316.025518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1316.053277] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1316.597284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1316.658626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1316.832801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1316.849704] LNet: 83245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.853453] LNet: 83245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1317.926765] LNet: Removed LNI 192.168.203.24@tcp [ 1317.968423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1318.058763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1318.097300] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1318.836045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1319.063641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1319.095900] LNetError: 83390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.101839] LNetError: 83390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1320.168749] LNet: Removed LNI 192.168.203.24@tcp [ 1320.192674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1320.261638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1320.284434] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1320.951336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1321.035731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1321.257684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1321.275583] LNet: 83537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.281597] LNet: 83537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1322.346798] LNet: Removed LNI 192.168.203.24@tcp [ 1322.378987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1322.425487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1322.449754] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1323.286168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1323.511817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1323.540814] LNetError: 83685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.550671] LNetError: 83685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1323.579854] LNet: Removed LNI 192.168.203.24@tcp [ 1323.611402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1323.700902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1323.755613] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1324.405076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1324.473612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1324.653388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1325.749218] LNet: Removed LNI 192.168.203.24@tcp [ 1325.779413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.865207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1325.886434] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1326.568676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1326.851324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1327.911198] LNet: Removed LNI 192.168.203.24@tcp [ 1327.929925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1327.978156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1328.010195] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1328.658732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1328.739847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1328.984443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1330.086808] LNet: Removed LNI 192.168.203.24@tcp [ 1330.116135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.164749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1330.187916] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1330.848411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1331.082221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1331.100568] LNet: 84275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.109320] LNet: 84275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1332.135653] LNet: Removed LNI 192.168.203.24@tcp [ 1332.151578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.190093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1332.210188] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1332.723874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1332.790578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1332.966834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1332.983154] LNetError: 84423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.990474] LNetError: 84423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1333.015260] LNet: Removed LNI 192.168.203.24@tcp [ 1333.034653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.077141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1333.103289] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1333.677490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1333.919526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1334.950942] LNet: Removed LNI 192.168.203.24@tcp [ 1334.975755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.043455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1335.064418] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1335.631834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1335.678542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1335.705325] LNet: Removed LNI 192.168.203.24@tcp [ 1335.728657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.788120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1335.817668] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1336.503492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1336.559291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1336.581075] LNet: Removed LNI 192.168.203.24@tcp [ 1336.597966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.660604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1336.734789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1336.770389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.817654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1336.892957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1336.941437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.001385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1337.084819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1337.120548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.165479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1337.181665] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1337.789353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1337.822345] LNet: Removed LNI 192.168.203.24@tcp [ 1337.837443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.897028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1342.972606] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:50:30 (1788749430) [ 1343.133418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1343.163945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1343.213981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1343.250799] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/180] [ 1344.131511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1344.166674] LNet: Removed LNI 192.168.203.24@tcp231 [ 1344.179748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1344.244867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1344.269338] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/180] [ 1344.275384] LNet: 85735:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1344.627194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1344.659941] LNet: Removed LNI 192.168.203.24@tcp231 [ 1344.680365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1344.705775] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/180] [ 1345.502885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1345.523712] LNet: Removed LNI 192.168.203.24@tcp231 [ 1345.535581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.590194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1345.606671] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/0] [ 1345.611109] LNet: 85887:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1345.862633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1345.887888] LNet: Removed LNI 192.168.203.24@tcp231 [ 1345.899883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1345.929295] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/0] [ 1346.612671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1346.635877] LNet: Removed LNI 192.168.203.24@tcp231 [ 1346.656517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.701223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1346.721373] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/60] [ 1346.724647] LNet: 86038:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1346.967758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1346.990205] LNet: Removed LNI 192.168.203.24@tcp231 [ 1347.005760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1347.029120] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/60] [ 1347.763347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1347.779280] LNet: 86181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.783252] LNet: 86181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 1347.799345] LNet: Removed LNI 192.168.203.24@tcp231 [ 1347.813779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1347.863305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1347.882637] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/180] [ 1348.115333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1348.144409] LNet: Removed LNI 192.168.203.24@tcp231 [ 1348.165711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1348.204467] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/180] [ 1353.466828] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:50:40 (1788749440) [ 1353.540956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1353.559480] LNetError: 86895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.564853] LNetError: 86895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1353.578331] LNet: Removed LNI 192.168.203.24@tcp231 [ 1353.602939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1353.638857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1353.654777] LNet: Added LNI 192.168.203.24@tcp232 [8/256/0/180] [ 1354.113185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1354.151784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1354.176797] LNet: Removed LNI 192.168.203.24@tcp232 [ 1354.195959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1354.231360] LNet: Added LNI 192.168.203.24@tcp232 [8/256/0/180] [ 1354.706970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1354.732990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1354.762378] LNet: Removed LNI 192.168.203.24@tcp232 [ 1354.775741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1354.794743] LNet: Added LNI 192.168.203.24@tcp232 [8/256/0/180] [ 1355.405382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1355.443633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1355.471033] LNet: Removed LNI 192.168.203.24@tcp232 [ 1355.488119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1355.525273] LNet: Added LNI 192.168.203.24@tcp232 [8/256/0/180] [ 1356.135851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1356.170897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1356.202239] LNet: Removed LNI 192.168.203.24@tcp232 [ 1356.219270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1360.568443] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:50:47 (1788749447) [ 1361.322464] Key type .llcrypt unregistered [ 1361.324720] Key type ._llcrypt unregistered [ 1365.800619] Key type ._llcrypt registered [ 1365.805665] Key type .llcrypt registered [ 1365.955839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1365.964464] alg: No test for adler32 (adler32-zlib) [ 1366.839536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1366.858989] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1371.780315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1377.869827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1377.898976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1377.914335] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1378.891829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_interrupt -r 2 [ 1378.916980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 2 [ 1378.955467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_interrupt -r 2 [ 1378.997679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 2 [ 1379.049074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_interrupt -r 2 [ 1379.078549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 2 [ 1379.108249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_interrupt -r 2 [ 1379.133882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 2 [ 1379.158148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1379.175879] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1379.180854] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1379.348731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1379.371307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1379.404952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1379.429768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1379.456538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1379.486407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1379.506727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1379.529868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1379.560975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1379.574087] LNet: 88688:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1379.585187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1379.610599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1379.630067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1379.653716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1379.678806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1379.702561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1380.078153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_dropped -r 2 [ 1380.108984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 2 [ 1380.142848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_dropped -r 2 [ 1380.173655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 2 [ 1380.196461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_dropped -r 2 [ 1380.236943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 2 [ 1380.268917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_dropped -r 2 [ 1380.301500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 2 [ 1380.336810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1380.353479] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1380.520979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1380.549551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1380.583848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1380.622847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1380.660713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1380.698656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1380.724967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1380.753674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1380.785785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1380.799843] LNet: 88811:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1380.812838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1380.837734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1380.871624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1380.903920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1380.927876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1380.960335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1381.352532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_aborted -r 2 [ 1381.380338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 2 [ 1381.410730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_aborted -r 2 [ 1381.446720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 2 [ 1381.485574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_aborted -r 2 [ 1381.526792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 2 [ 1381.567843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_aborted -r 2 [ 1381.600432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 2 [ 1381.628750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1381.641941] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1381.794928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1381.826150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1381.846866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1381.868950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1381.899361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1381.932508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1381.963562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1382.007872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1382.034239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1382.049907] LNet: 88933:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1382.064587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1382.088962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1382.108468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.129868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1382.152781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1382.178748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1382.581395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_no_route -r 2 [ 1382.605652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 2 [ 1382.630846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_no_route -r 2 [ 1382.660314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 2 [ 1382.693655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_no_route -r 2 [ 1382.724435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 2 [ 1382.751543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_no_route -r 2 [ 1382.776536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 2 [ 1382.804604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1382.980512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1382.997328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1383.024203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1383.050669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1383.082373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1383.109384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1383.133823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1383.165744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1383.202602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1383.245931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1383.276664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1383.309421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1383.344128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1383.378528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1383.408556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1383.807364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_timeout -r 2 [ 1383.842213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 2 [ 1383.880492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_timeout -r 2 [ 1383.924050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 2 [ 1383.968877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_timeout -r 2 [ 1384.015819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 2 [ 1384.056380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_timeout -r 2 [ 1384.087879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 2 [ 1384.125458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1384.146278] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1384.150940] LNet: Skipped 1 previous similar message [ 1384.318186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1384.342622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1384.369863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1384.395559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1384.418230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1384.443859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1384.468969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1384.492933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1384.517468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1384.533527] LNet: 89177:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1384.538410] LNet: 89177:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1384.557671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1384.579603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1384.606893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1384.629230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1384.653936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1384.688242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1385.101426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_dropped -r 2 [ 1385.133089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 2 [ 1385.164429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_dropped -r 2 [ 1385.190870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 2 [ 1385.228653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_dropped -r 2 [ 1385.259572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 2 [ 1385.287381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_dropped -r 2 [ 1385.316664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 2 [ 1385.340526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1385.358381] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.124@tcp [ 1385.540225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1385.575710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1385.605064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1385.626397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1385.652368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1385.681989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1385.706105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1385.734973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1385.762908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1385.787527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1385.814358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1385.842546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1385.863884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1385.893948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1385.918351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1389.374290] LNet: 89373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.379321] LNetError: 89373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.432145] LNet: Removed LNI 192.168.203.24@tcp [ 1390.436221] LNetError: 89373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.451328] LNet: Removed LNI 192.168.203.24@tcp1 [ 1390.770703] Key type .llcrypt unregistered [ 1390.774732] Key type ._llcrypt unregistered [ 1403.332212] Key type ._llcrypt registered [ 1403.337932] Key type .llcrypt registered [ 1404.259725] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:51:31 (1788749491) [ 1404.516455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1404.525570] alg: No test for adler32 (adler32-zlib) [ 1405.404298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.461764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1405.481775] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1406.124588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1406.141877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.24@tcp [ 1406.157489] LNetError: 90350:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.24@tcp to recovery [ 1406.207931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.24@tcp [ 1406.296632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1410.826161] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:51:37 (1788749497) [ 1410.909856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1410.925884] LNet: 90934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.933693] LNetError: 90934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.946469] LNet: Removed LNI 192.168.203.24@tcp [ 1410.976849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.030380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1411.046248] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1411.677471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1411.694812] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1412.353031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.24@tcp [ 1412.369893] LNetError: 91205:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.24@tcp to recovery [ 1413.088377] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1413.090427] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1413.552287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.24@tcp [ 1420.030608] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:51:47 (1788749507) [ 1421.388754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg324-server [ 1422.645653] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:51:49 (1788749509) [ 1423.952295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg324-server [ 1425.199972] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:51:52 (1788749512) [ 1426.664904] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg324-server [ 1428.194342] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:51:55 (1788749515) [ 1428.263096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.295828] LNet: 92828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.303128] LNetError: 92828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.310290] LNet: Removed LNI 192.168.203.24@tcp [ 1428.324347] LNet: Removed LNI 192.168.203.24@tcp1 [ 1428.347497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.405941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1428.423534] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1429.023728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1429.067233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1429.108316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1429.181647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1429.210527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1429.261239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1433.959767] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:52:01 (1788749521) [ 1434.165280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.184207] LNet: 93566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.194089] LNetError: 93566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.198193] LNetError: 93566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1434.210177] LNet: Removed LNI 192.168.203.24@tcp [ 1434.226900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.291860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.309982] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1439.367720] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:52:06 (1788749526) [ 1439.424772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1439.441522] LNet: 94262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.446324] LNetError: 94262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.462441] LNet: Removed LNI 192.168.203.24@tcp [ 1439.480985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.533897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1439.558671] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1445.909241] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:52:12 (1788749532) [ 1447.865344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1450.477627] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:52:16 (1788749536) [ 1451.091166] LNet: 95304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.097569] LNetError: 95304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.118285] LNet: Removed LNI 192.168.203.24@tcp [ 1452.178950] Key type .llcrypt unregistered [ 1452.180620] Key type ._llcrypt unregistered [ 1460.361987] Key type ._llcrypt registered [ 1460.368881] Key type .llcrypt registered [ 1460.636040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1460.654715] alg: No test for adler32 (adler32-zlib) [ 1461.511179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1461.548327] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1472.091981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1478.710729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.124@tcp [ 1479.366885] LNet: 95721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.386536] LNetError: 95721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.414631] LNet: Removed LNI 192.168.203.24@tcp [ 1480.040735] Key type .llcrypt unregistered [ 1480.045376] Key type ._llcrypt unregistered [ 1493.930108] Key type ._llcrypt registered [ 1493.932374] Key type .llcrypt registered [ 1495.345623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1496.957213] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1498.714710] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:53:05 (1788749585) [ 1500.181788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1501.971711] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:53:08 (1788749588) [ 1503.804931] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg324-server [ 1505.555932] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:53:12 (1788749592) [ 1507.107493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg324-server [ 1508.813270] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:53:15 (1788749595) [ 1510.555262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1510.595870] alg: No test for adler32 (adler32-zlib) [ 1511.733578] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1512.061117] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1513.824677] Key type lgssc registered [ 1515.189557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1536.612200] Key type lgssc unregistered [ 1536.900795] LNet: 98892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.908631] LNetError: 98892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.926146] LNet: Removed LNI 192.168.203.24@tcp [ 1537.674494] Key type .llcrypt unregistered [ 1537.681708] Key type ._llcrypt unregistered [ 1552.084475] Key type ._llcrypt registered [ 1552.091818] Key type .llcrypt registered [ 1553.283595] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:54:00 (1788749640) [ 1553.744869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1553.761564] alg: No test for adler32 (adler32-zlib) [ 1554.685889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.734068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1554.777987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1554.840300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1554.883276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1561.040551] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:54:07 (1788749647) [ 1561.144643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.166365] LNet: 100298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.200313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.292892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.312348] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1562.339643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.353799] LNet: 100448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.363680] LNetError: 100448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.385751] LNet: Removed LNI 192.168.203.24@tcp [ 1562.397236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.456459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1562.488503] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1562.978896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.001843] LNetError: 100476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.014982] LNet: Removed LNI 192.168.203.24@tcp [ 1563.035219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.166450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1563.193457] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1570.189676] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:54:16 (1788749656) [ 1570.356886] LNet: 101061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.364109] LNet: 101061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1570.369826] LNetError: 101061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.386721] LNet: Removed LNI 192.168.203.24@tcp [ 1571.059541] Key type .llcrypt unregistered [ 1571.063357] Key type ._llcrypt unregistered [ 1577.009342] Key type ._llcrypt registered [ 1577.013823] Key type .llcrypt registered [ 1577.241735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1577.259886] alg: No test for adler32 (adler32-zlib) [ 1578.253505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.364215] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1578.367657] LNet: *** cfs_fail_loc=e003, val=100*** [ 1579.308954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1579.348121] LNet: *** cfs_fail_loc=e003, val=100*** [ 1579.358936] LNet: Skipped 2 previous similar messages [ 1579.366624] Ping reply received with mapping: [ 1579.372831] found metadata reply NI[192.168.203.24@tcp] [ 1579.376465] 18 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1579.379771] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1579.384687] 5a 5a 5a 5a 5a 5a 5a 5a [ 1579.570730] LNet: 101460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.581558] LNetError: 101460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.603865] LNet: Removed LNI 192.168.203.24@tcp [ 1580.244496] Key type .llcrypt unregistered [ 1580.248623] Key type ._llcrypt unregistered [ 1594.560621] Key type ._llcrypt registered [ 1594.567657] Key type .llcrypt registered [ 1596.087513] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:54:42 (1788749682) [ 1596.728338] Key type .llcrypt unregistered [ 1596.731267] Key type ._llcrypt unregistered [ 1603.854984] Key type ._llcrypt registered [ 1603.860382] Key type .llcrypt registered [ 1604.138644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1604.155819] alg: No test for adler32 (adler32-zlib) [ 1605.193471] LNetError: 102497:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1605.507640] Key type .llcrypt unregistered [ 1605.510865] Key type ._llcrypt unregistered [ 1611.049324] Key type ._llcrypt registered [ 1611.052678] Key type .llcrypt registered [ 1612.384838] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:54:59 (1788749699) [ 1612.868628] Key type .llcrypt unregistered [ 1612.873751] Key type ._llcrypt unregistered [ 1618.144971] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1618.149175] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1620.590969] Key type ._llcrypt registered [ 1620.593390] Key type .llcrypt registered [ 1620.841319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1620.872325] alg: No test for adler32 (adler32-zlib) [ 1621.818331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.870800] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1630.298681] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:55:17 (1788749717) [ 1630.483528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.498553] LNet: 104270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.504848] LNetError: 104270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1630.529238] LNet: Removed LNI 10.1.2.3@tcp [ 1630.550113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.155458] Key type .llcrypt unregistered [ 1631.160200] Key type ._llcrypt unregistered [ 1636.903478] Key type ._llcrypt registered [ 1636.907781] Key type .llcrypt registered [ 1637.130820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1637.161658] alg: No test for adler32 (adler32-zlib) [ 1638.089292] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1638.146410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1638.334816] LNet: 104555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.348402] LNetError: 104555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.371286] LNet: Removed LNI 192.168.203.24@tcp [ 1638.973661] Key type .llcrypt unregistered [ 1638.979145] Key type ._llcrypt unregistered [ 1648.599862] Key type ._llcrypt registered [ 1648.602398] Key type .llcrypt registered [ 1649.729563] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:55:36 (1788749736) [ 1650.370286] Key type .llcrypt unregistered [ 1650.376526] Key type ._llcrypt unregistered [ 1656.337458] Key type ._llcrypt registered [ 1656.343542] Key type .llcrypt registered [ 1656.563325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1656.576377] alg: No test for adler32 (adler32-zlib) [ 1663.763827] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:55:50 (1788749750) [ 1663.926547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.997492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.056816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1664.079865] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1664.720867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1664.761587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1664.796719] LNet: Removed LNI 192.168.203.24@tcp [ 1664.979932] LNet: 106294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.384545] Key type .llcrypt unregistered [ 1665.386302] Key type ._llcrypt unregistered [ 1675.613899] Key type ._llcrypt registered [ 1675.618487] Key type .llcrypt registered [ 1677.071501] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:56:03 (1788749763) [ 1677.423607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.447122] alg: No test for adler32 (adler32-zlib) [ 1678.331696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.387787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.408802] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1678.930950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1678.964389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.24@tcp [ 1678.983829] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1678.986832] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1678.989283] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1678, 0, 0 [ 1678.992390] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1678.994986] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1678.999748] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1679.006169] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1678, 0, 0 [ 1679.009922] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1679.012077] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1679.014361] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1679.016793] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1678, 0, 1728 [ 1679.024061] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1679.027053] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1679.030476] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1679.033396] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1678, 0, 1728 [ 1679.038837] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1685.209548] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:56:11 (1788749771) [ 1685.526256] LNet: 107832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.531391] LNetError: 107832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.551878] LNet: Removed LNI 192.168.203.24@tcp [ 1686.233591] Key type .llcrypt unregistered [ 1686.236553] Key type ._llcrypt unregistered [ 1692.171756] Key type ._llcrypt registered [ 1692.182497] Key type .llcrypt registered [ 1692.428944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1692.462313] alg: No test for adler32 (adler32-zlib) [ 1693.379637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1693.419048] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1701.553390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1711.021944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.108814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1711.141054] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1712.159163] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1712.339588] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1712.346617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1712.355317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1714.689815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1714.716394] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1725.853582] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:56:52 (1788749812) [ 1726.099768] LNet: 109327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.126184] LNetError: 109327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.136168] LNet: Removed LNI 192.168.203.24@tcp [ 1727.144101] LNetError: 109327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.155308] LNet: Removed LNI 192.168.203.24@tcp1 [ 1727.172080] LNet: Removed LNI 10.1.2.3@tcp99 [ 1727.985769] Key type .llcrypt unregistered [ 1727.987666] Key type ._llcrypt unregistered [ 1733.821039] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1733.829838] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1736.388911] Key type ._llcrypt registered [ 1736.392341] Key type .llcrypt registered [ 1736.632789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1736.646726] alg: No test for adler32 (adler32-zlib) [ 1737.580348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.647827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1737.698086] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1738.386281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1738.427703] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1740.615235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1740.665236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1740.721150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1740.772898] LNetError: 110076:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1740.822979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1740.894949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1741.278882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1741.356841] LNet: 110096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.369617] LNetError: 110096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1741.378381] LNet: Removed LNI 10.1.2.3@tcp [ 1742.441099] LNet: Removed LNI 192.168.203.24@tcp [ 1743.125550] Key type .llcrypt unregistered [ 1743.130741] Key type ._llcrypt unregistered [ 1756.893740] Key type ._llcrypt registered [ 1756.896852] Key type .llcrypt registered [ 1758.795149] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:57:24 (1788749844) [ 1759.272765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.283755] alg: No test for adler32 (adler32-zlib) [ 1760.308099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.375993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1760.430587] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1762.090062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1762.264264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg324-client@tcp [ 1770.670868] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:57:36 (1788749856) [ 1770.947382] LNet: 111629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.978790] LNetError: 111629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.030629] LNet: Removed LNI 192.168.203.24@tcp [ 1771.945407] Key type .llcrypt unregistered [ 1771.957493] Key type ._llcrypt unregistered [ 1778.985558] Key type ._llcrypt registered [ 1778.989807] Key type .llcrypt registered [ 1779.245506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1779.282343] alg: No test for adler32 (adler32-zlib) [ 1780.333480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1780.374716] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1780.418545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1789.582316] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:57:55 (1788749875) [ 1791.336848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1793.447339] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:57:59 (1788749879) [ 1795.326960] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1797.251438] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:58:03 (1788749883) [ 1797.318635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.355121] LNet: 113046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.370581] LNetError: 113046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.426114] LNet: Removed LNI 192.168.203.24@tcp [ 1797.447289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.503933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1797.522696] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1798.240295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1798.300691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1806.103807] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:58:12 (1788749892) [ 1806.231185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.281534] LNet: 113754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.298107] LNetError: 113754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.322063] LNet: Removed LNI 192.168.203.24@tcp [ 1806.401617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.480355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1806.557736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1807.513651] Key type .llcrypt unregistered [ 1807.516223] Key type ._llcrypt unregistered [ 1818.633348] Key type ._llcrypt registered [ 1818.636833] Key type .llcrypt registered [ 1820.181342] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:58:26 (1788749906) [ 1820.566961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.584200] alg: No test for adler32 (adler32-zlib) [ 1821.605719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.683455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.724966] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1822.658600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1822.744754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1830.132290] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:58:36 (1788749916) [ 1830.284415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.320505] LNet: 115306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.335089] LNetError: 115306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.364852] LNet: Removed LNI 192.168.203.24@tcp [ 1830.432288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.493383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1830.516216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1837.520562] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:58:43 (1788749923) [ 1837.746313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1837.815969] LNet: 115894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.841462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.911842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1838.465992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1838.519386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1838.595307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1839.149564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1839.197901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1839.307986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1839.340714] LNet: 115928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.371306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.501148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1840.188901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1840.270481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1840.343386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1840.864115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1840.899500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1848.528511] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:58:54 (1788749934) [ 1848.650502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1848.708246] LNet: 116522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.755797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.826141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1848.843322] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1849.751769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1849.774710] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1850.638265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1850.708294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1850.928816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.959447] LNetError: 116809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.970587] LNet: Removed LNI 192.168.203.24@tcp [ 1850.984862] LNet: Removed LNI 192.168.203.24@tcp2 [ 1851.009259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.077814] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1851.085321] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1851.093340] LNet: 116815:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1851.831847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1852.054967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1852.080410] LNetError: 116852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.095689] LNetError: 116852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1852.115273] LNet: Removed LNI 192.168.203.24@tcp [ 1852.143592] LNet: Removed LNI 192.168.203.24@tcp2 [ 1852.195128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.244229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.265614] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1853.328499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1853.389334] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1854.596232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1854.679332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1854.756367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1855.083945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1855.119483] LNet: 117136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.131395] LNet: 117136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1855.146223] LNetError: 117136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.157356] LNetError: 117136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1855.175759] LNet: Removed LNI 192.168.203.24@tcp [ 1856.233845] LNet: Removed LNI 192.168.203.24@tcp2 [ 1856.278828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.410192] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1856.419919] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1856.431335] LNet: 117143:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1857.097799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1857.408609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1857.444764] LNet: Removed LNI 192.168.203.24@tcp [ 1858.478236] LNet: Removed LNI 192.168.203.24@tcp2 [ 1858.513586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.556464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1858.573928] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1859.568385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1859.598810] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1860.426384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1860.506787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1860.769049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.810847] LNetError: 117459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.818617] LNetError: 117459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1860.826778] LNet: Removed LNI 192.168.203.24@tcp [ 1860.847427] LNet: Removed LNI 192.168.203.24@tcp2 [ 1860.899293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.965217] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1860.971463] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1860.975886] LNet: 117465:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1861.631687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1861.850835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1862.880212] LNet: Removed LNI 192.168.203.24@tcp [ 1862.897167] LNet: Removed LNI 192.168.203.24@tcp2 [ 1862.962338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.069460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1863.110359] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1864.136453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1864.172171] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1865.328730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1865.374988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1865.469882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1865.794625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1865.814775] LNet: 117787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.824173] LNet: 117787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1866.849409] LNet: Removed LNI 192.168.203.24@tcp [ 1867.880082] LNet: Removed LNI 192.168.203.24@tcp2 [ 1867.904628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.007032] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1868.015289] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1868.024100] LNet: 117794:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1868.947891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1869.206794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.226120] LNetError: 117831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.230517] LNetError: 117831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1869.235197] LNet: Removed LNI 192.168.203.24@tcp [ 1870.251832] LNet: Removed LNI 192.168.203.24@tcp2 [ 1870.303861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.382385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1870.428994] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1871.524283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1871.549226] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1872.395842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1872.426918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1872.480582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1872.851612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.888298] LNet: Removed LNI 192.168.203.24@tcp [ 1873.912787] LNet: Removed LNI 192.168.203.24@tcp2 [ 1873.966851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1874.057963] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1874.067952] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1874.079668] LNet: 118134:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1874.804750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1875.089675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.129574] LNet: Removed LNI 192.168.203.24@tcp [ 1876.150461] LNet: Removed LNI 192.168.203.24@tcp2 [ 1876.207322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.293714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1876.318031] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1877.330437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1877.358631] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1878.622526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1878.700927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1878.764966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1878.846852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1879.478036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.546132] LNet: Removed LNI 192.168.203.24@tcp [ 1881.587616] LNet: Removed LNI 192.168.203.24@tcp2 [ 1881.636461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.725127] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1881.737633] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1882.457940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1891.739767] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1893.727961] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:59:40 (1788749980) [ 1893.971069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1894.020072] LNet: 119253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.037398] LNet: 119253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1894.060455] LNetError: 119253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.075532] LNetError: 119253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1894.093463] LNet: Removed LNI 192.168.203.24@tcp [ 1895.144115] LNet: Removed LNI 192.168.203.24@tcp2 [ 1895.226894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.786726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1895.852889] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1897.159942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1897.199567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1898.825553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.871861] LNet: Removed LNI 10.1.2.3@tcp [ 1898.895773] LNet: Removed LNI 192.168.203.24@tcp [ 1898.946270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1899.182473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1899.229293] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1899.238733] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1899.986691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1900.024386] LNet: Removed LNI 192.168.203.24@tcp [ 1900.038506] LNet: Removed LNI 10.1.2.3@tcp [ 1908.880740] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:59:55 (1788749995) [ 1909.035929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.119631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.215964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1909.296594] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1910.352241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1910.385570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1910.425426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1910.454602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1910.480674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1910.517809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1910.561611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1910.595876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1910.640900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1910.700845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1910.762812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1910.804642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1911.011451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1918.882949] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 23:00:04 (1788750004) [ 1919.197823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.324951] LNet: Removed LNI 192.168.203.24@tcp [ 1920.388261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.504889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1920.553985] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1921.569496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 1921.628092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1921.693969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1921.753382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1921.794414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1921.846421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1921.896839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1921.935887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1921.973915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.001563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.039357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.069541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.271188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1922.354760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.424230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.469282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.538085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.576250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.640699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.711479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.770989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.856738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1922.936560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1931.186287] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 23:00:17 (1788750017) [ 1931.307486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1931.349827] LNet: 121622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.362228] LNet: 121622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1931.385177] LNetError: 121622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.397379] LNetError: 121622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1931.449786] LNet: Removed LNI 192.168.203.24@tcp [ 1931.539274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1931.755464] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1932.021588] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1932.038701] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1932.044117] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1934.487792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1934.553113] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1935.684613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1935.731700] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1938.462316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1938.509346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1938.597521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.24@tcp -m PUT -e local_timeout -r 1 [ 1938.684312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1938.805596] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1938.818258] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1939.131494] LNetError: 121691:0:(lib-move.c:4794:lnet_parse()) 192.168.203.24@tcp, src 192.168.203.24@tcp: Dropping PUT (error -108 looking up sender) [ 1941.217243] LNet: Removed LNI 10.1.2.3@tcp [ 1942.246914] LNet: Removed LNI 192.168.203.24@tcp [ 1942.674298] Key type .llcrypt unregistered [ 1942.680639] Key type ._llcrypt unregistered [ 1957.559413] Key type ._llcrypt registered [ 1957.561737] Key type .llcrypt registered [ 1958.887452] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 23:00:45 (1788750045) [ 1959.543584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1959.560345] alg: No test for adler32 (adler32-zlib) [ 1960.493941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1961.049575] LNet: 123000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.633488] Key type .llcrypt unregistered [ 1961.635460] Key type ._llcrypt unregistered [ 1969.013605] Key type ._llcrypt registered [ 1969.020287] Key type .llcrypt registered [ 1969.243420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1969.252493] alg: No test for adler32 (adler32-zlib) [ 1970.261157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1970.305510] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1978.741229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1978.990805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1979.645745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.124@tcp [ 1979.864306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1979.895345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1979.965568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1979.999542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1980.039823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1980.099457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1980.146404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1980.193333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1980.240515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1980.265413] LNet: 123462:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1980.284525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1980.324718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1980.360374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1980.431752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1980.486758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1980.527912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1981.204927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1981.459967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1981.493791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1981.531695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1981.550884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1981.601543] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.124@tcp [ 1981.602235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1981.678743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1981.739372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1981.790168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1981.822922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1981.855148] LNet: 123581:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1981.880324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1981.912523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1981.936933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1981.962465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1981.987268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1982.034302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1982.811803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.124@tcp [ 1983.249144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1983.279266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1983.307084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1983.336722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1983.367708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1983.398473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1983.430812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1983.465342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1983.497497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1983.519874] LNet: 123701:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1983.537511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1983.567446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1983.595249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1983.627904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1983.664168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1983.700450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1984.527460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1984.846698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1984.922228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1984.975731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1985.034832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1985.077726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1985.126637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1985.204765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1985.252571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1985.322372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1985.362899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1985.416359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1985.472832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1985.510089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1985.587835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1985.645877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1986.036144] LNet: 123866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.047811] LNetError: 123866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.068335] LNet: Removed LNI 192.168.203.24@tcp [ 1986.621316] Key type .llcrypt unregistered [ 1986.625821] Key type ._llcrypt unregistered [ 2003.066112] Key type ._llcrypt registered [ 2003.067978] Key type .llcrypt registered [ 2004.556102] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 23:01:31 (1788750091) [ 2005.098046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2005.119185] alg: No test for adler32 (adler32-zlib) [ 2006.163084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.700318] LNet: 124751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.528962] Key type .llcrypt unregistered [ 2007.538679] Key type ._llcrypt unregistered [ 2013.511060] Key type ._llcrypt registered [ 2013.514234] Key type .llcrypt registered [ 2013.715484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2013.725669] alg: No test for adler32 (adler32-zlib) [ 2014.718136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2014.740731] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2022.475737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 2032.236154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.284894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2032.328747] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 2033.323941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2033.969725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.124@tcp [ 2034.176065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2034.207580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2034.209298] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.124@tcp [ 2034.240943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2034.278538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2034.306411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2034.335712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2034.366852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2034.423673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2034.476787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2034.496434] LNet: 125402:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2034.541355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2034.608679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2034.659354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2034.722810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2034.782613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2034.828220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2035.406331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2035.669181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2035.712409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2035.806196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2035.834920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2035.863909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2035.889647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2035.917391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2035.957106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2035.994134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2036.021850] LNet: 125521:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2036.046787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2036.090357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2036.129285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2036.183509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2036.234568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2036.275353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2037.011224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.124@tcp [ 2037.290313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2037.325784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2037.361402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2037.398376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2037.447978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2037.512987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2037.560577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2037.603388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2037.647182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2037.687919] LNet: 125640:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2037.712745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2037.756036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2037.815777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2037.862168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2037.920827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2037.980041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2038.646345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2038.900708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2038.931078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2038.966777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2039.011912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2039.051359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2039.079830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2039.106952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2039.138944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2039.186585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2039.258088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2039.324659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2039.382726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2039.415710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2039.474249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2039.510803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2045.049503] LNet: 125841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.065720] LNetError: 125841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.080286] LNet: Removed LNI 192.168.203.24@tcp [ 2045.104790] LNet: Removed LNI 192.168.203.24@tcp1 [ 2045.525881] Key type .llcrypt unregistered [ 2045.527989] Key type ._llcrypt unregistered [ 2062.788194] Key type ._llcrypt registered [ 2062.791506] Key type .llcrypt registered [ 2064.359642] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 23:02:30 (1788750150) [ 2064.805653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2064.827853] alg: No test for adler32 (adler32-zlib) [ 2065.769825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2065.868785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2065.922417] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2066.646341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2066.683408] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2067.555160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2067.627712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2068.704535] LNet: Removed LNI 192.168.203.24@tcp [ 2075.198733] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 23:02:41 (1788750161) [ 2076.851887] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2078.656140] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 23:02:45 (1788750165) [ 2086.382989] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 23:02:53 (1788750173) [ 2093.863220] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 23:03:00 (1788750180) [ 2101.138885] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 23:03:07 (1788750187) [ 2109.132357] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 23:03:15 (1788750195) [ 2118.272921] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 23:03:24 (1788750204) [ 2127.055846] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 23:03:33 (1788750213) [ 2135.003421] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 23:03:41 (1788750221) [ 2141.950402] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 23:03:48 (1788750228) [ 2148.897140] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 23:03:55 (1788750235) [ 2155.285315] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 23:04:02 (1788750242) [ 2163.136537] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 23:04:09 (1788750249) [ 2169.937183] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 23:04:16 (1788750256) [ 2175.816797] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 23:04:22 (1788750262) [ 2182.839472] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 23:04:29 (1788750269) [ 2189.340165] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 23:04:36 (1788750276) [ 2195.733770] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 23:04:42 (1788750282) [ 2202.824811] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 23:04:49 (1788750289) [ 2202.924595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2202.952972] LNet: 137524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.963604] LNetError: 137524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.989336] LNet: Removed LNI 192.168.203.24@tcp2 [ 2203.030239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.176714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2203.209494] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2204.487267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2204.513743] LNet: 137674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.527373] LNetError: 137674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.544404] LNet: Removed LNI 192.168.203.24@tcp [ 2204.570750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2204.639952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2204.663838] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2213.338859] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 23:04:59 (1788750299) [ 2220.777920] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 23:05:07 (1788750307) [ 2227.319248] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 23:05:14 (1788750314) [ 2234.761418] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 23:05:21 (1788750321) [ 2241.793966] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 23:05:28 (1788750328) [ 2250.465556] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 23:05:36 (1788750336) [ 2260.362774] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 23:05:46 (1788750346) [ 2269.439467] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 23:05:55 (1788750355) [ 2277.935862] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 23:06:04 (1788750364) [ 2285.511915] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 23:06:12 (1788750372) [ 2293.564734] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 23:06:20 (1788750380) [ 2301.001486] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 23:06:27 (1788750387) [ 2309.180513] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2082 sec ========== 23:06:35 (1788750395) [ 2309.483492] LNet: 145140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.487253] LNetError: 145140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.498949] LNet: Removed LNI 192.168.203.24@tcp [ 2309.971585] Key type .llcrypt unregistered [ 2309.973476] Key type ._llcrypt unregistered [ 2324.089798] Key type ._llcrypt registered [ 2324.096956] Key type .llcrypt registered [ 2324.312704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2324.323720] alg: No test for adler32 (adler32-zlib) [ 2325.438811] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2325.807208] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2327.576362] Key type lgssc registered [ 2329.115989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2398.008648] Lustre: Mounted lustre-client - version 2.17.58_41_g2290de3 [ 2402.810662] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2410.249742] Lustre: Unmounted lustre-client [ 2439.674119] Key type lgssc unregistered [ 2440.035201] LNet: 148104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.049477] LNetError: 148104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.080673] LNet: Removed LNI 192.168.203.24@tcp [ 2440.991288] Key type .llcrypt unregistered [ 2440.997411] Key type ._llcrypt unregistered