[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 440834015 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002388] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009141] LSM: Security Framework initializing [ 0.010059] Yama: becoming mindful. [ 0.011044] SELinux: Initializing. [ 0.012067] *** VALIDATE selinux *** [ 0.020682] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025783] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027078] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] *** VALIDATE tmpfs *** [ 0.031141] *** VALIDATE proc *** [ 0.032258] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.034266] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036059] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039025] Speculative Store Bypass: Vulnerable [ 0.041987] debug: unmapping init [mem 0xffffffffbb259000-0xffffffffbb260fff] [ 0.043917] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044682] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045027] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052307] rcu: Hierarchical SRCU implementation. [ 0.054349] smp: Bringing up secondary CPUs ... [ 0.055566] x86: Booting SMP configuration: [ 0.056025] .... node #0, CPUs: #1 #2 #3 [ 0.060011] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138182] node 0 deferred pages initialised in 72ms [ 0.142139] devtmpfs: initialized [ 0.143241] x86/mm: Memory block size: 128MB [ 0.145776] gcov: version magic: 0x41383552 [ 0.147285] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149218] pinctrl core: initialized pinctrl subsystem [ 0.150208] [ 0.150815] ************************************************************* [ 0.151020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152011] ** ** [ 0.153017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154014] ** ** [ 0.155016] ** This means that this kernel is built to expose internal ** [ 0.156014] ** IOMMU data structures, which may compromise security on ** [ 0.157013] ** your system. ** [ 0.158019] ** ** [ 0.159015] ** If you see this message and you are not debugging the ** [ 0.160015] ** kernel, report this immediately to your vendor! ** [ 0.161014] ** ** [ 0.162018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163014] ************************************************************* [ 0.164668] NET: Registered protocol family 16 [ 0.165480] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168450] cpuidle: using governor menu [ 0.170266] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173579] PCI: Using configuration type 1 for base access [ 0.175136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.184108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.185037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186193] cryptd: max_cpu_qlen set to 1000 [ 0.188069] ACPI: Added _OSI(Module Device) [ 0.189011] ACPI: Added _OSI(Processor Device) [ 0.190007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.195046] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198531] ACPI: Interpreter enabled [ 0.199060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200012] ACPI: Using IOAPIC for interrupt routing [ 0.201093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202380] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.211233] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.212043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.213023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.214081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.216282] acpiphp: Slot [2] registered [ 0.217138] acpiphp: Slot [5] registered [ 0.218156] acpiphp: Slot [6] registered [ 0.219111] acpiphp: Slot [3] registered [ 0.220107] acpiphp: Slot [4] registered [ 0.221101] acpiphp: Slot [7] registered [ 0.222134] acpiphp: Slot [8] registered [ 0.223122] acpiphp: Slot [9] registered [ 0.224100] acpiphp: Slot [10] registered [ 0.225108] acpiphp: Slot [11] registered [ 0.226109] acpiphp: Slot [12] registered [ 0.227095] acpiphp: Slot [13] registered [ 0.228142] acpiphp: Slot [14] registered [ 0.229053] acpiphp: Slot [15] registered [ 0.230106] acpiphp: Slot [16] registered [ 0.232104] acpiphp: Slot [17] registered [ 0.234159] acpiphp: Slot [18] registered [ 0.235104] acpiphp: Slot [19] registered [ 0.236097] acpiphp: Slot [20] registered [ 0.237119] acpiphp: Slot [21] registered [ 0.238110] acpiphp: Slot [22] registered [ 0.239156] acpiphp: Slot [23] registered [ 0.241104] acpiphp: Slot [24] registered [ 0.243101] acpiphp: Slot [25] registered [ 0.244134] acpiphp: Slot [26] registered [ 0.246103] acpiphp: Slot [27] registered [ 0.247135] acpiphp: Slot [28] registered [ 0.249095] acpiphp: Slot [29] registered [ 0.250085] acpiphp: Slot [30] registered [ 0.252089] acpiphp: Slot [31] registered [ 0.253130] PCI host bridge to bus 0000:00 [ 0.255057] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.258054] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.260019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.263022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.265017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.268020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.269186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.271940] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.275363] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.282317] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.286049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.289020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.291013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.293014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.295479] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.297828] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.300041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.302804] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.307012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.316013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.321014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.326871] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.335012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.340938] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.351930] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.360091] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.372013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.379014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.398013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.410348] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.412409] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.415465] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.418428] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.421220] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.426226] iommu: Default domain type: Passthrough [ 0.429170] SCSI subsystem initialized [ 0.430144] ACPI: bus type USB registered [ 0.432184] usbcore: registered new interface driver usbfs [ 0.434070] usbcore: registered new interface driver hub [ 0.435074] usbcore: registered new device driver usb [ 0.437180] pps_core: LinuxPPS API ver. 1 registered [ 0.439011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.442072] PTP clock support registered [ 0.444122] EDAC MC: Ver: 3.0.0 [ 0.445447] PCI: Using ACPI for IRQ routing [ 0.447729] NetLabel: Initializing [ 0.449013] NetLabel: domain hash size = 128 [ 0.450009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.452080] NetLabel: unlabeled traffic allowed by default [ 0.454140] vgaarb: loaded [ 0.456168] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.458012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.464831] clocksource: Switched to clocksource kvm-clock [ 0.567172] VFS: Disk quotas dquot_6.6.0 [ 0.568829] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.571515] *** VALIDATE ramfs *** [ 0.572488] *** VALIDATE hugetlbfs *** [ 0.574192] pnp: PnP ACPI init [ 0.576324] pnp: PnP ACPI: found 6 devices [ 0.626219] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.630464] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.632813] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.635126] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.637866] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.640514] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.643728] NET: Registered protocol family 2 [ 0.646505] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.651479] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.655101] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.660250] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.663394] TCP: Hash tables configured (established 65536 bind 65536) [ 0.666360] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.669071] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.672036] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.674862] NET: Registered protocol family 1 [ 0.677596] RPC: Registered named UNIX socket transport module. [ 0.680232] RPC: Registered udp transport module. [ 0.682336] RPC: Registered tcp transport module. [ 0.684213] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.686247] NET: Registered protocol family 44 [ 0.687754] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.689589] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.691283] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.693037] PCI: CLS 0 bytes, default 64 [ 0.694228] Unpacking initramfs... [ 2.086782] debug: unmapping init [mem 0xffff91b97cc64000-0xffff91b97ffcffff] [ 2.090892] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.093310] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.096483] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.573286] Initialise system trusted keyrings [ 2.574686] Key type blacklist registered [ 2.576405] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.584771] zbud: loaded [ 2.587393] *** VALIDATE nfs *** [ 2.588461] *** VALIDATE nfs4 *** [ 2.590417] pstore: using deflate compression [ 2.593392] Platform Keyring initialized [ 2.693805] NET: Registered protocol family 38 [ 2.695199] Key type asymmetric registered [ 2.696883] Asymmetric key parser 'x509' registered [ 2.698842] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.701944] io scheduler mq-deadline registered [ 2.703878] io scheduler kyber registered [ 2.705690] io scheduler bfq registered [ 2.708027] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.711130] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.714287] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.717166] ACPI: Power Button [PWRF] [ 2.722531] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.729642] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.740073] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.769598] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.799843] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.805189] Non-volatile memory driver v1.3 [ 2.807135] Linux agpgart interface v0.103 [ 2.844255] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.848424] vda: detected capacity change from 0 to 76775424 [ 2.864397] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.867582] vdb: detected capacity change from 0 to 1073741824 [ 2.875326] libphy: Fixed MDIO Bus: probed [ 2.880478] usbcore: registered new interface driver usbserial_generic [ 2.883232] usbserial: USB Serial support registered for generic [ 2.885844] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.890305] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.892034] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.894489] mousedev: PS/2 mouse device common for all mice [ 2.897215] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.899528] rtc_cmos 00:05: RTC can wake from S4 [ 2.903506] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.904511] rtc_cmos 00:05: registered as rtc0 [ 2.908625] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.909520] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.915542] intel_pstate: CPU model not supported [ 2.918661] hid: raw HID events driver (C) Jiri Kosina [ 2.920772] usbcore: registered new interface driver usbhid [ 2.923076] usbhid: USB HID core driver [ 2.924885] drop_monitor: Initializing network drop monitor service [ 2.927202] Initializing XFRM netlink socket [ 2.928876] NET: Registered protocol family 10 [ 2.932812] Segment Routing with IPv6 [ 2.934468] NET: Registered protocol family 17 [ 2.936648] mpls_gso: MPLS GSO support [ 2.942782] RAS: Correctable Errors collector initialized. [ 2.944818] AVX version of gcm_enc/dec engaged. [ 2.946433] AES CTR mode by8 optimization enabled [ 3.024620] sched_clock: Marking stable (3024591248, 0)->(3971580479, -946989231) [ 3.028127] registered taskstats version 1 [ 3.029869] Loading compiled-in X.509 certificates [ 3.032404] zswap: loaded using pool lzo/zbud [ 3.059195] Key type big_key registered [ 3.071262] Key type encrypted registered [ 3.073093] ima: No TPM chip found, activating TPM-bypass! [ 3.077846] ima: Allocated hash algorithm: sha1 [ 3.079648] ima: No architecture policies found [ 3.081621] evm: Initialising EVM extended attributes: [ 3.083716] evm: security.selinux [ 3.085501] evm: security.ima [ 3.086619] evm: security.capability [ 3.088025] evm: HMAC attrs: 0x1 [ 3.091196] rtc_cmos 00:05: setting system clock to 2026-09-09 05:04:13 UTC (1788930253) [ 3.096974] debug: unmapping init [mem 0xffffffffbc203000-0xffffffffbc3fffff] [ 3.100016] debug: unmapping init [mem 0xffffffffbaf82000-0xffffffffbb258fff] [ 3.109068] Write protecting the kernel read-only data: 28672k [ 3.113918] debug: unmapping init [mem 0xffffffffb9603000-0xffffffffb97fffff] [ 3.116321] debug: unmapping init [mem 0xffffffffb9f14000-0xffffffffb9ffffff] [ 3.148673] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.154867] systemd[1]: Detected virtualization kvm. [ 3.156463] systemd[1]: Detected architecture x86-64. [ 3.157732] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.182217] systemd[1]: No hostname configured. [ 3.183559] systemd[1]: Set hostname to . [ 3.185821] random: systemd: uninitialized urandom read (16 bytes read) [ 3.187899] systemd[1]: Initializing machine ID from random generator. [ 3.303432] random: systemd: uninitialized urandom read (16 bytes read) [ 3.305497] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.308475] random: systemd: uninitialized urandom read (16 bytes read) [ 3.309981] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.312628] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.840502] device-mapper: uevent: version 1.0.3 [ 3.843147] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 4.441594] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.500113] virtio_net virtio0 ens2: renamed from eth0 [ 4.512721] scsi host0: ata_piix [ 4.529605] scsi host1: ata_piix [ 4.542515] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.545501] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.732183] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.595487] random: crng init done [ 9.597317] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.775764] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.830726] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.081690] SELinux: Disabled at runtime. [ 11.140157] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.147784] systemd[1]: Detected virtualization kvm. [ 11.149695] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.631441] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.634975] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.643895] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.647842] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.651304] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.658336] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.663051] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2[ 11.757802] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.121135] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.381332] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.438701] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.507566] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.528641] EDAC sbridge: Ver: 1.1.2 [ 13.482905] Key type dns_resolver registered [ 13.792425] NFS: Registering the id_resolver key type [ 13.794586] Key type id_resolver registered [ 13.796584] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg309-client login: [ 73.935444] libcfs: loading out-of-tree module taints kernel. [ 74.115704] Key type ._llcrypt registered [ 74.126063] Key type .llcrypt registered [ 75.214426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 75.229329] alg: No test for adler32 (adler32-zlib) [ 76.861950] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 78.103143] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 79.984249] Key type lgssc registered [ 82.579313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 238.257614] Lustre: Mounted lustre-client - version 2.17.58_39_gf1a8369 [ 243.669987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.827609] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing check_logdir /tmp/testlogs/ [ 261.180645] hrtimer: interrupt took 3711470 ns [ 263.228923] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing yml_node [ 264.161119] Lustre: lustre-OST0000-osc-ffff91b9c9e0b000: disconnect after 23s idle [ 268.313716] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 271.030926] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 273.526487] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 275.236840] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 9 01:08:44 EDT 2026 [ 291.643244] Lustre: Unmounted lustre-client [ 331.642749] Key type lgssc unregistered [ 331.907867] LNet: 8422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.918360] LNetError: 8422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 331.938434] LNet: Removed LNI 192.168.203.9@tcp [ 332.834341] Key type .llcrypt unregistered [ 332.836500] Key type ._llcrypt unregistered [ 360.886390] Key type ._llcrypt registered [ 360.888789] Key type .llcrypt registered [ 361.147111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 361.161184] alg: No test for adler32 (adler32-zlib) [ 362.047387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 362.116428] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 362.134990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 366.627122] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 366.752172] LNet: 9294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.763094] LNetError: 9294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 366.790183] LNet: Removed LNI 192.168.203.9@tcp [ 367.606450] Key type .llcrypt unregistered [ 367.614097] Key type ._llcrypt unregistered [ 373.775193] Key type ._llcrypt registered [ 373.780899] Key type .llcrypt registered [ 375.147395] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:10:24 (1788930624) [ 375.606378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 375.624400] alg: No test for adler32 (adler32-zlib) [ 376.560903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.665454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.685959] LNet: 9710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.701623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.744934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 376.765713] LNet: 9717:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 383.739539] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:10:32 (1788930632) [ 383.951699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.968852] LNet: 10295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.989659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.078691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 391.740752] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:10:40 (1788930640) [ 391.937741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.989549] LNet: 10874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.053344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.242088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 401.354760] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:10:49 (1788930649) [ 401.533642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.585774] LNet: 11464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.661579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.787275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 410.624746] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:10:59 (1788930659) [ 410.723756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.743963] LNet: 12045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.790157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.924846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 411.405681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 420.012950] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:11:08 (1788930668) [ 420.400378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.425645] LNet: 12643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.452299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.542711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 421.026952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 421.471122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 429.446924] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:11:18 (1788930678) [ 429.558104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.642583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.714369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 436.837131] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:11:25 (1788930685) [ 437.270828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.310371] LNet: 13825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.317355] LNet: 13825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 437.335822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.398900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 437.459970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 437.862435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 437.908705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 438.408439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 438.457571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 438.986729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 439.026150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 439.248500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 439.272243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 439.482958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 439.502896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 440.001492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 440.055126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 440.568619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 440.620548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 441.084178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 441.110855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 441.543502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 441.598784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 441.975848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 442.007077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 442.269692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 442.307747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 442.783191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 442.827937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 449.824340] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:11:38 (1788930698) [ 450.013401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.069620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.125453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 450.165049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 457.700715] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:11:46 (1788930706) [ 457.848683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.905511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.014247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 458.043904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 465.562332] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:11:54 (1788930714) [ 465.745317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.817533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.950545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 466.008241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 474.033275] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:12:02 (1788930722) [ 474.475791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.508657] LNet: 16265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.521400] LNet: 16265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 474.588343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.710836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 474.760846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 482.671490] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:12:11 (1788930731) [ 483.086399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.129491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.227424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 483.274238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 491.421279] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:12:20 (1788930740) [ 491.746766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.784423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.897356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 491.958434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 499.832738] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:12:28 (1788930748) [ 500.042894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.079238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.202885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 500.736526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 508.129101] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:12:37 (1788930757) [ 508.550027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.609225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.841220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 509.087327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 516.862221] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:12:45 (1788930765) [ 517.065404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.138798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.296721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 517.590935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 525.566710] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:12:54 (1788930774) [ 525.910229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.967399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.374733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 526.666540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 534.089537] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:13:02 (1788930782) [ 534.378120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.444755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.554993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 534.611809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 541.339216] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:13:10 (1788930790) [ 541.618597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.651420] LNet: 21099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.655914] LNet: 21099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 541.675870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.778481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 541.857698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 550.116343] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:13:18 (1788930798) [ 550.560704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.648617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.780855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 558.758464] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:13:27 (1788930807) [ 559.381916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.481853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.589988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 567.452592] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:13:36 (1788930816) [ 567.684123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.788078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.948632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 568.764136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 576.810298] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:13:45 (1788930825) [ 577.029648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.101629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.220115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 577.266386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 585.226867] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:13:54 (1788930834) [ 585.661905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.714368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.806430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 585.847930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 593.488964] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:14:02 (1788930842) [ 593.718508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.797216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.919448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 593.974971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 601.532306] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:14:10 (1788930850) [ 601.635701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.707139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.779626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 601.811854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 609.202213] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:14:18 (1788930858) [ 609.525991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.569340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.627726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 609.690957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 609.712816] LNetError: 25825:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 609.752978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 609.787186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 617.048842] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:14:26 (1788930866) [ 617.678565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.745634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.838732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BYO8AtJzx3 [ 617.938306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BYO8AtJzx3 [ 617.989256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BYO8AtJzx3 [ 625.006318] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:14:33 (1788930873) [ 625.208358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.264977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.310061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 625.352826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 625.418938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 632.092877] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:14:41 (1788930881) [ 632.431859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.503542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.587211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 632.613050] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 634.033990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 635.669045] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 637.552234] Key type lgssc registered [ 638.807433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 642.562036] LNetError: 27587:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.9@tcp99 has no available nets [ 645.373626] Key type lgssc unregistered [ 645.810795] LNetError: 28693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.851357] LNet: Removed LNI 192.168.203.9@tcp [ 646.709360] Key type .llcrypt unregistered [ 646.714684] Key type ._llcrypt unregistered [ 654.248740] Key type ._llcrypt registered [ 654.250987] Key type .llcrypt registered [ 655.804904] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:15:04 (1788930904) [ 656.548654] Key type .llcrypt unregistered [ 656.554062] Key type ._llcrypt unregistered [ 664.172511] Key type ._llcrypt registered [ 664.176578] Key type .llcrypt registered [ 664.422727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 664.457089] alg: No test for adler32 (adler32-zlib) [ 665.462902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.582774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 665.611023] LNet: 29604:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 665.618533] LNet: 29604:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 665.623189] LNet: 29604:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 665.633508] LNet: Message forwarding will be enabled in 2s [ 668.704124] LNet: Message forwarding enabled [ 669.287553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 669.336960] LNet: Message forwarding will stop in 2s to 6s [ 672.803701] LNet: Message forwarding disabled [ 673.976018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 674.009873] LNet: 29703:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 674.022980] LNet: 29703:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 674.030365] LNet: 29703:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 674.067336] LNet: Message forwarding will be enabled in 2s [ 674.346591] LNet: 29716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.083520] Key type .llcrypt unregistered [ 675.086601] Key type ._llcrypt unregistered [ 689.673492] Key type ._llcrypt registered [ 689.677964] Key type .llcrypt registered [ 691.138783] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:15:39 (1788930939) [ 691.723787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 691.740233] alg: No test for adler32 (adler32-zlib) [ 692.709449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.777825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 700.585230] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:15:48 (1788930948) [ 700.725231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.777609] LNet: 31125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.817251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.904920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 700.978992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 701.043185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 701.099783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 701.159972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 701.215223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 701.285702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 701.357161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 701.441941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 701.501530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 701.567735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 701.618287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 701.710050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 701.771779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 701.824832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 701.911969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 701.972454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 702.025539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 702.052391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 702.076825] LNetError: 31174:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 709.003421] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:15:58 (1788930958) [ 709.431620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.457754] LNet: 31745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.497807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.569617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 717.782317] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:16:06 (1788930966) [ 719.949818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg309-server [ 722.304822] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:16:11 (1788930971) [ 722.733621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.777164] LNet: 32678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.836494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.914345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.942749] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 723.980649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[10-14]@tcp [ 723.999217] LNet: 32811:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 724.092367] LNetError: 32682:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.12@tcp is being used as a gateway but routing feature is not turned on [ 724.119181] LNetError: 32682:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.12@tcp has gone from up to down [ 731.924867] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:16:20 (1788930980) [ 733.830166] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg309-server [ 735.931281] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:16:24 (1788930984) [ 736.207683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.264338] LNet: 33747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.286196] LNetError: 33747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.307475] LNet: Removed LNI 192.168.203.9@tcp [ 736.380374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.477771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 736.518511] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 737.582099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[10-15/2]@tcp [ 737.631254] LNet: 33864:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 737.641224] LNet: 33864:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 737.667699] LNetError: 33751:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.14@tcp is being used as a gateway but routing feature is not turned on [ 737.685414] LNetError: 33751:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 737.685926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[10-15/2]@tcp [ 737.693093] LNetError: 33751:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.14@tcp has gone from up to down [ 737.718723] LNetError: 33751:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 746.035639] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:16:34 (1788930994) [ 746.349924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.377487] LNet: 34456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.387915] LNetError: 34456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 747.431449] LNet: Removed LNI 192.168.203.9@tcp [ 747.534946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.662332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 747.747618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.801968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.886472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 747.956864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.003540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.084356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 748.189589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.254128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.311405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 748.413569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.465499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.516610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 748.593761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.643295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.746854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 748.881883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.941167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.106956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 749.202876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.237605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.367649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 749.496141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.538293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.622925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 749.762795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.827654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.958491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 750.111982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.214058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.283870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 750.342779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.379219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.473422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 758.351745] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:16:47 (1788931007) [ 760.959937] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg309-server [ 763.385444] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:16:51 (1788931011) [ 765.794157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg309-server [ 768.076940] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:16:56 (1788931016) [ 768.185663] LNet: 35850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.192137] LNet: 35850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 769.181526] Key type .llcrypt unregistered [ 769.191419] Key type ._llcrypt unregistered [ 778.015855] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 778.258055] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 778.278023] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 778.287935] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 781.064208] Key type ._llcrypt registered [ 781.067830] Key type .llcrypt registered [ 781.353665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 781.373978] alg: No test for adler32 (adler32-zlib) [ 782.418924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.510879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.565985] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 783.581600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 783.620627] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 786.282714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 786.320706] LNet: Removed LNI 10.1.2.3@tcp [ 786.543203] LNet: 36579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.564096] LNetError: 36579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.596057] LNet: Removed LNI 192.168.203.9@tcp [ 787.390401] Key type .llcrypt unregistered [ 787.394634] Key type ._llcrypt unregistered [ 800.808653] Key type ._llcrypt registered [ 800.817242] Key type .llcrypt registered [ 802.176926] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:17:31 (1788931051) [ 802.960281] Key type .llcrypt unregistered [ 802.963159] Key type ._llcrypt unregistered [ 810.259871] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 810.425951] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 810.444116] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 810.467209] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 813.036848] Key type ._llcrypt registered [ 813.039098] Key type .llcrypt registered [ 813.311572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 813.330482] alg: No test for adler32 (adler32-zlib) [ 814.255127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.320874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.365574] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 815.288626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 815.309545] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.259379] LNet: 38068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.267625] LNetError: 38068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 818.274529] LNet: Removed LNI 10.1.2.3@tcp [ 818.319373] LNet: Removed LNI 192.168.203.9@tcp [ 819.054027] Key type .llcrypt unregistered [ 819.057279] Key type ._llcrypt unregistered [ 836.650442] Key type ._llcrypt registered [ 836.657974] Key type .llcrypt registered [ 838.336532] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:18:06 (1788931086) [ 839.085055] Key type .llcrypt unregistered [ 839.086642] Key type ._llcrypt unregistered [ 847.541805] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 848.027158] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.039803] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 848.042205] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 849.890460] Key type ._llcrypt registered [ 849.897437] Key type .llcrypt registered [ 850.233962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 850.245549] alg: No test for adler32 (adler32-zlib) [ 851.370256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.474323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 851.532833] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 854.890885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 854.927834] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 857.922625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 857.969947] LNet: Removed LNI 10.1.2.3@tcp [ 858.010233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 858.044490] LNet: Removed LNI 10.1.2.31@tcp [ 858.067463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 858.087037] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 859.066192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 859.089072] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 859.804613] LNet: 39978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.809105] LNetError: 39978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 859.814479] LNet: Removed LNI 10.1.2.31@tcp [ 859.826582] LNet: Removed LNI 10.1.2.3@tcp [ 860.297378] Key type .llcrypt unregistered [ 860.301676] Key type ._llcrypt unregistered [ 875.833389] Key type ._llcrypt registered [ 875.836989] Key type .llcrypt registered [ 877.272047] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:18:46 (1788931126) [ 877.718706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.737489] alg: No test for adler32 (adler32-zlib) [ 878.681980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.733276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 878.772318] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 879.708385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.751395] LNet: 40929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.759938] LNetError: 40929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.779477] LNet: Removed LNI 192.168.203.9@tcp [ 879.810218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.882301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.9@tcp [ 879.907979] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 880.757152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.9@tcp [ 880.801902] LNet: Removed LNI 192.168.203.9@tcp [ 880.936873] LNet: 41064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.642354] Key type .llcrypt unregistered [ 881.645668] Key type ._llcrypt unregistered [ 895.239585] Key type ._llcrypt registered [ 895.242132] Key type .llcrypt registered [ 896.848263] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:19:05 (1788931145) [ 899.713660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg309-server [ 902.028505] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:19:10 (1788931150) [ 902.805248] Key type .llcrypt unregistered [ 902.810103] Key type ._llcrypt unregistered [ 910.665335] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 911.038563] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 911.041878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 911.070975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 914.010646] Key type ._llcrypt registered [ 914.013685] Key type .llcrypt registered [ 914.334397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 914.371095] alg: No test for adler32 (adler32-zlib) [ 915.495489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.669672] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 917.162047] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 921.042370] LNet: 42897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.071264] LNetError: 42897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.077761] LNet: Removed LNI 192.168.203.9@tcp1 [ 921.094642] LNet: Removed LNI 10.1.2.3@tcp2 [ 922.129534] Key type .llcrypt unregistered [ 922.131803] Key type ._llcrypt unregistered [ 938.776560] Key type ._llcrypt registered [ 938.782333] Key type .llcrypt registered [ 940.444703] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:19:48 (1788931188) [ 940.897310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.907258] alg: No test for adler32 (adler32-zlib) [ 941.883770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.960418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 942.014262] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 943.051509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.084279] LNet: 43865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.089766] LNetError: 43865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.111654] LNet: Removed LNI 192.168.203.9@tcp [ 943.145939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.220792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 943.239551] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 950.382541] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:19:59 (1788931199) [ 950.626167] LNet: 44455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.633927] LNetError: 44455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.658692] LNet: Removed LNI 192.168.203.9@tcp [ 951.373629] Key type .llcrypt unregistered [ 951.378495] Key type ._llcrypt unregistered [ 958.161292] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.425071] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 958.430023] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 958.436198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.168885] Key type ._llcrypt registered [ 961.172973] Key type .llcrypt registered [ 961.457387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.484907] alg: No test for adler32 (adler32-zlib) [ 962.446277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.531824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.568529] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 963.364203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 963.410040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.935205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.969715] LNet: 45195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.986281] LNetError: 45195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.999196] LNet: Removed LNI 10.1.2.3@tcp [ 966.023567] LNet: Removed LNI 192.168.203.9@tcp [ 966.069492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.153113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 966.207203] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 966.226461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 966.875776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.910523] LNet: 45223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.924882] LNetError: 45223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 966.934039] LNetError: 45223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 966.943805] LNet: Removed LNI 10.1.2.3@tcp [ 966.957852] LNet: Removed LNI 192.168.203.9@tcp [ 966.983874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.050681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 967.087649] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 967.097843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 967.670178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 967.687993] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 968.781347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.798766] LNet: 45385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.805506] LNetError: 45385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 968.813111] LNetError: 45385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 968.822097] LNet: Removed LNI 10.1.2.3@tcp [ 968.828622] LNet: Removed LNI 192.168.203.9@tcp [ 968.846701] LNet: Removed LNI 192.168.203.9@tcp2 [ 968.874511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.965573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 969.011105] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 969.018416] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 969.024595] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 976.989279] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:20:25 (1788931225) [ 977.177326] LNet: 45977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.186715] LNetError: 45977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 977.198579] LNetError: 45977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 977.207949] LNet: Removed LNI 10.1.2.3@tcp [ 977.209683] LNet: Removed LNI 192.168.203.9@tcp [ 977.224402] LNet: Removed LNI 192.168.203.9@tcp2 [ 978.152944] Key type .llcrypt unregistered [ 978.158893] Key type ._llcrypt unregistered [ 985.066921] Key type ._llcrypt registered [ 985.071380] Key type .llcrypt registered [ 985.265818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 985.284857] alg: No test for adler32 (adler32-zlib) [ 986.216455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.284326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 986.316611] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 987.277430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.310966] LNet: 46374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.323556] LNetError: 46374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.364338] LNet: Removed LNI 192.168.203.9@tcp [ 987.400773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.475804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 987.496021] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 987.944487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.962535] LNet: 46404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.968221] LNetError: 46404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.985229] LNet: Removed LNI 192.168.203.9@tcp [ 988.010105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.117227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 988.155476] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 988.762930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.792755] LNet: Removed LNI 192.168.203.9@tcp [ 988.826932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.947500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 989.004971] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 989.741751] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 989.992615] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 990.006817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 990.023332] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 992.332852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.351933] LNet: 46509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.357183] LNet: 46509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 992.362644] LNetError: 46509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.369474] LNetError: 46509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 992.392791] LNet: Removed LNI 192.168.203.9@tcp [ 992.411128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.476945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 992.522332] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1000.458809] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:20:49 (1788931249) [ 1000.763946] LNet: 47106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.776529] LNetError: 47106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.812504] LNet: Removed LNI 192.168.203.9@tcp [ 1001.552645] Key type .llcrypt unregistered [ 1001.559232] Key type ._llcrypt unregistered [ 1009.183871] Key type ._llcrypt registered [ 1009.186236] Key type .llcrypt registered [ 1009.409332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.439310] alg: No test for adler32 (adler32-zlib) [ 1010.401560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.508636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1010.553444] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1011.485646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.524449] LNet: 47502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.530772] LNetError: 47502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.546825] LNet: Removed LNI 192.168.203.9@tcp [ 1011.574529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.667728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1011.732846] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1019.498335] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:21:08 (1788931268) [ 1019.851716] LNet: 48097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.859826] LNetError: 48097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.887635] LNet: Removed LNI 192.168.203.9@tcp [ 1020.671576] Key type .llcrypt unregistered [ 1020.677965] Key type ._llcrypt unregistered [ 1028.515209] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1028.632220] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1028.636312] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1028.640124] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1031.248339] Key type ._llcrypt registered [ 1031.250148] Key type .llcrypt registered [ 1031.411256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1031.429433] alg: No test for adler32 (adler32-zlib) [ 1032.415307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1032.485857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1032.558968] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1033.577637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1033.616944] LNet: 48545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.636984] LNetError: 48545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.692978] LNet: Removed LNI 192.168.203.9@tcp [ 1033.738619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1033.826343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1033.869118] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1041.245699] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:21:30 (1788931290) [ 1041.665699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.710344] LNet: 49143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.717894] LNetError: 49143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.734687] LNet: Removed LNI 192.168.203.9@tcp [ 1041.772563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.881735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1048.871134] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:21:37 (1788931297) [ 1049.045968] LNet: 49723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.632496] Key type .llcrypt unregistered [ 1049.634929] Key type ._llcrypt unregistered [ 1056.658351] Key type ._llcrypt registered [ 1056.660534] Key type .llcrypt registered [ 1056.790577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1056.800549] alg: No test for adler32 (adler32-zlib) [ 1057.693993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.751073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1057.771610] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1058.500751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.522695] LNet: 50118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.527531] LNetError: 50118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.542378] LNet: Removed LNI 192.168.203.9@tcp [ 1058.579258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.632339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1058.662235] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1065.659951] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:21:54 (1788931314) [ 1065.810285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1065.850501] LNet: 50707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.865838] LNetError: 50707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.899164] LNet: Removed LNI 192.168.203.9@tcp [ 1065.941591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.022322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1072.612121] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:22:01 (1788931321) [ 1074.380477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1076.241889] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:22:05 (1788931325) [ 1078.060276] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1079.943621] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:22:08 (1788931328) [ 1081.623621] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1083.346264] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:22:12 (1788931332) [ 1085.196643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1087.248574] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:22:15 (1788931335) [ 1088.889383] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1091.115668] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:22:19 (1788931339) [ 1091.323876] LNet: 52731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.056656] Key type .llcrypt unregistered [ 1092.061158] Key type ._llcrypt unregistered [ 1098.555455] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1098.843951] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1098.846972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1098.851477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1101.692803] Key type ._llcrypt registered [ 1101.696083] Key type .llcrypt registered [ 1101.953428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1101.977785] alg: No test for adler32 (adler32-zlib) [ 1102.916383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.031506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1103.080938] LNet: Added LNI 192.168.203.9@tcp [8/512/0/180] [ 1103.092740] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1103.094835] LNet: 53055:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1103.308102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1103.338762] LNet: 53074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.345029] LNetError: 53074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1103.350228] LNet: Removed LNI 10.1.2.3@tcp [ 1103.352777] LNetError: 53074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.375955] LNet: Removed LNI 192.168.203.9@tcp [ 1103.394553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.475860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1103.493552] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1103.563893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1103.588666] LNet: 53099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.611586] LNetError: 53099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.625941] LNet: Removed LNI 192.168.203.9@tcp [ 1103.657017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.753689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1103.786294] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1103.794728] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1103.796468] LNet: 53107:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1103.926802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1103.972216] LNet: 53125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.999726] LNetError: 53125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1104.013405] LNet: Removed LNI 10.1.2.3@tcp [ 1104.020504] LNetError: 53125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.040059] LNet: Removed LNI 192.168.203.9@tcp [ 1104.093276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.251899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1104.299281] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1104.307878] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1104.314307] LNet: 53133:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1104.588817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1104.621147] LNet: 53159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.626274] LNetError: 53159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1104.638106] LNet: Removed LNI 10.1.2.3@tcp [ 1104.640946] LNetError: 53159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.661538] LNet: Removed LNI 192.168.203.9@tcp [ 1104.679725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.759072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1104.796378] LNet: Added LNI 192.168.203.9@tcp [8/256/0/182] [ 1104.801916] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1104.968966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1104.991168] LNet: 53187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.006673] LNetError: 53187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1105.017676] LNet: Removed LNI 10.1.2.3@tcp [ 1105.021241] LNetError: 53187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.032760] LNet: Removed LNI 192.168.203.9@tcp [ 1105.058907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.141910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1105.178706] LNet: Added LNI 192.168.203.9@tcp [8/256/0/182] [ 1105.184622] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1105.186664] LNet: 53195:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1105.307185] LNet: Removed LNI 10.1.2.3@tcp [ 1105.320902] LNet: Removed LNI 192.168.203.9@tcp [ 1105.943261] Key type .llcrypt unregistered [ 1105.950481] Key type ._llcrypt unregistered [ 1119.988370] Key type ._llcrypt registered [ 1119.990840] Key type .llcrypt registered [ 1121.409689] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:22:50 (1788931370) [ 1122.925466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1125.040425] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:22:53 (1788931373) [ 1125.536393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1125.560533] alg: No test for adler32 (adler32-zlib) [ 1126.542781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.609707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1126.627252] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1127.408073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.9 [ 1127.426523] LNet: 54456:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1127.449667] LNet: Removed LNI 192.168.203.9@tcp [ 1135.247880] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:23:03 (1788931383) [ 1135.500598] LNet: 55024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.528181] LNetError: 55024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.548836] LNet: Removed LNI 192.168.203.9@tcp [ 1136.360103] Key type .llcrypt unregistered [ 1136.364108] Key type ._llcrypt unregistered [ 1143.654679] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1143.980847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1146.731691] Key type ._llcrypt registered [ 1146.737808] Key type .llcrypt registered [ 1146.952207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1146.963754] alg: No test for adler32 (adler32-zlib) [ 1148.433087] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1155.997618] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:23:24 (1788931404) [ 1156.409386] LNet: 55937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.417202] LNetError: 55937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1156.445653] LNet: Removed LNI 10.1.2.3@tcp [ 1157.179769] Key type .llcrypt unregistered [ 1157.185697] Key type ._llcrypt unregistered [ 1163.938990] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1164.280669] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1164.283736] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1164.291917] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1165.894693] Key type ._llcrypt registered [ 1165.896861] Key type .llcrypt registered [ 1166.004975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1166.024677] alg: No test for adler32 (adler32-zlib) [ 1167.055894] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1173.974778] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:23:43 (1788931423) [ 1175.568934] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1177.431819] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:23:46 (1788931426) [ 1177.750834] LNet: 57123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.764815] LNetError: 57123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1177.802781] LNet: Removed LNI 10.1.2.3@tcp [ 1178.603408] Key type .llcrypt unregistered [ 1178.606586] Key type ._llcrypt unregistered [ 1187.377659] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1187.858062] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1187.862257] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1187.877077] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1191.016498] Key type ._llcrypt registered [ 1191.022047] Key type .llcrypt registered [ 1191.411616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.461804] alg: No test for adler32 (adler32-zlib) [ 1192.504356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1192.641656] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1202.905804] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:24:11 (1788931451) [ 1203.489824] LNet: 58350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.495479] LNetError: 58350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1203.509328] LNet: Removed LNI 10.1.2.3@tcp [ 1204.108377] Key type .llcrypt unregistered [ 1204.113912] Key type ._llcrypt unregistered [ 1211.439320] Key type ._llcrypt registered [ 1211.443856] Key type .llcrypt registered [ 1211.621181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1211.630239] alg: No test for adler32 (adler32-zlib) [ 1212.519887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1212.535981] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1220.470933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1221.057762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_interrupt -r 1 [ 1221.096305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_interrupt -r 1 [ 1221.139031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1221.168462] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1221.359851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1221.387102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1221.420782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1221.446772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.493381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1221.538764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.578345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.619152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.644149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.663183] LNet: 58788:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.683261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.740442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.782469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.813391] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1221.856841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.893742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.934718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.989679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1222.776864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_dropped -r 1 [ 1222.828469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_dropped -r 1 [ 1222.879301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1222.915500] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1223.210788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1223.252410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1223.287790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1223.340083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1223.390107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1223.435359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1223.479425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1223.532317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1223.573090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1223.589391] LNet: 58899:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1223.608644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1223.648622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1223.695812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1223.743869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1223.781266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1223.847688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1223.897870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1224.396447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_aborted -r 1 [ 1224.433382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_aborted -r 1 [ 1224.468993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1224.493858] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1224.832039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.884779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1224.919056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.963193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.005237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1225.054951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.096643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.129863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.163435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.180574] LNet: 59010:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.197540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1225.224208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1225.267334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.308751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.338723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.373898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.409860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1226.073132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_no_route -r 1 [ 1226.125438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_no_route -r 1 [ 1226.178638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1226.395709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.436176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.460980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.487391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.521647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1226.563605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.597320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.638812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.677734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.710850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.766656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.806219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.831624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.870945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.920681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.966303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1227.591735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_timeout -r 1 [ 1227.643515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_timeout -r 1 [ 1227.699258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1227.734375] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1227.743396] LNet: Skipped 1 previous similar message [ 1227.966948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.996227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.035790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.069905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.109451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1228.134826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.153574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.172520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.195087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.214393] LNet: 59233:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.222457] LNet: 59233:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1228.248065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.280449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.314369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.342857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.367434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.405674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.452915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1228.964280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_error -r 1 [ 1228.996658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_error -r 1 [ 1229.028640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1229.208660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.246240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.292296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.324802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.361716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1229.383931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.413574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.445852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.496978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.545969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.586080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.607960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.645257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.709172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.766454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.814807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1230.080392] LNet: 59385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.091683] LNetError: 59385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.105852] LNet: Removed LNI 192.168.203.9@tcp [ 1230.405572] Key type .llcrypt unregistered [ 1230.408573] Key type ._llcrypt unregistered [ 1247.779116] Key type ._llcrypt registered [ 1247.781366] Key type .llcrypt registered [ 1249.128461] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:24:57 (1788931497) [ 1250.278850] Key type .llcrypt unregistered [ 1250.283239] Key type ._llcrypt unregistered [ 1257.157638] Key type ._llcrypt registered [ 1257.159722] Key type .llcrypt registered [ 1257.354839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1257.370292] alg: No test for adler32 (adler32-zlib) [ 1258.340739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1258.376168] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1265.420458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1277.106654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1277.158680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1277.181446] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1278.372818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_interrupt -r 1 [ 1278.399362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_interrupt -r 1 [ 1278.431681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_interrupt -r 1 [ 1278.462444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_interrupt -r 1 [ 1278.522608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_interrupt -r 1 [ 1278.562959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_interrupt -r 1 [ 1278.610797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_interrupt -r 1 [ 1278.649915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_interrupt -r 1 [ 1278.690724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1278.719740] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1278.727743] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1278.952950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.990625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.021409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.049641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.084956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1279.131292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1279.175490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1279.213460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1279.252780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1279.268817] LNet: 60852:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1279.283808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1279.314543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1279.352511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.401383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.431281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.465625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1279.509744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1280.073811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_dropped -r 1 [ 1280.132718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_dropped -r 1 [ 1280.168057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_dropped -r 1 [ 1280.192314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_dropped -r 1 [ 1280.235851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_dropped -r 1 [ 1280.271555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_dropped -r 1 [ 1280.307711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_dropped -r 1 [ 1280.356648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_dropped -r 1 [ 1280.406936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1280.438969] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1280.445971] LNet: Skipped 2 previous similar messages [ 1280.647882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.678902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1280.722199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.752482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.780870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1280.807801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.832601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.867413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.938711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.965782] LNet: 60975:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1281.002149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1281.038696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1281.071327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1281.095640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1281.121753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.153868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1281.181511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1281.817068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_aborted -r 1 [ 1281.860684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_aborted -r 1 [ 1281.903787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_aborted -r 1 [ 1281.961581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_aborted -r 1 [ 1282.008672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_aborted -r 1 [ 1282.046084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_aborted -r 1 [ 1282.114637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_aborted -r 1 [ 1282.163667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_aborted -r 1 [ 1282.223798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1282.243283] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1282.249188] LNet: Skipped 2 previous similar messages [ 1282.459803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1282.491976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1282.522921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1282.556682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1282.597557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1282.625070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1282.656624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1282.684900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1282.713993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1282.727530] LNet: 61098:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1282.740675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1282.764880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1282.790969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1282.823630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1282.846433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.885130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.920849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1283.465225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_no_route -r 1 [ 1283.513845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_no_route -r 1 [ 1283.606623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_no_route -r 1 [ 1283.658954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_no_route -r 1 [ 1283.701391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_no_route -r 1 [ 1283.761392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_no_route -r 1 [ 1283.843836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_no_route -r 1 [ 1283.904688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_no_route -r 1 [ 1283.960819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1284.216140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1284.241858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1284.270916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1284.301223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1284.327377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1284.355704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1284.402751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1284.432607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1284.468034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1284.547320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1284.584223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1284.627525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1284.682550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1284.722867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.770760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1284.824920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1285.471398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_timeout -r 1 [ 1285.522679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_timeout -r 1 [ 1285.586160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_timeout -r 1 [ 1285.634933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_timeout -r 1 [ 1285.708906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_timeout -r 1 [ 1285.769730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_timeout -r 1 [ 1285.828755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_timeout -r 1 [ 1285.884824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_timeout -r 1 [ 1285.920657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1285.957046] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1285.963874] LNet: Skipped 5 previous similar messages [ 1286.278460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1286.306608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1286.343296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1286.388959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1286.429516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1286.470743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1286.509530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1286.549121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1286.587357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1286.604899] LNet: 61345:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1286.611200] LNet: 61345:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1286.628840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1286.681777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1286.722223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1286.777255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1286.832688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1286.855885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1286.906747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1287.431474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_error -r 1 [ 1287.466696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_error -r 1 [ 1287.497696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_error -r 1 [ 1287.535474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_error -r 1 [ 1287.592866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_error -r 1 [ 1287.629875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_error -r 1 [ 1287.667498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_error -r 1 [ 1287.700524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_error -r 1 [ 1287.737783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1287.995710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.035403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.103759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.138945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.183565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1288.223924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.264706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.303670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.339576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.364946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.391991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.424584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.462813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.494104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.520222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.578760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1294.566743] LNet: 61545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.572619] LNetError: 61545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.575019] LNet: Removed LNI 192.168.203.9@tcp [ 1295.590694] LNet: Removed LNI 192.168.203.9@tcp1 [ 1296.061183] Key type .llcrypt unregistered [ 1296.063621] Key type ._llcrypt unregistered [ 1312.495715] Key type ._llcrypt registered [ 1312.500058] Key type .llcrypt registered [ 1313.768563] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:26:02 (1788931562) [ 1314.489597] Key type .llcrypt unregistered [ 1314.494822] Key type ._llcrypt unregistered [ 1320.922062] Key type ._llcrypt registered [ 1320.924043] Key type .llcrypt registered [ 1321.092944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1321.107746] alg: No test for adler32 (adler32-zlib) [ 1322.011614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1322.040245] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1329.848810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1330.670267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_dropped -r 1 [ 1330.724691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_dropped -r 1 [ 1330.790967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1330.813053] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1331.017322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1331.066668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1331.124298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1331.170286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1331.201456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1331.242926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1331.293099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1331.344149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1331.410456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1331.449916] LNet: 62792:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1331.491284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1331.539456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1331.580771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1331.628896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1331.667389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1331.709580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1331.740597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1332.327934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_error -r 1 [ 1332.357208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_error -r 1 [ 1332.382533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1332.429225] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1332.656395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1332.681740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1332.705438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1332.738452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1332.766673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1332.794378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1332.819545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1332.844647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1332.880373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1332.912976] LNet: 62903:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1332.928880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1332.965606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1332.995653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1333.032577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1333.075071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1333.107247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1333.155104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1333.701281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_timeout -r 1 [ 1333.737714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_timeout -r 1 [ 1333.771303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1333.799770] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1333.994918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1334.023675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1334.053373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1334.087128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1334.119769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1334.180365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1334.211721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1334.261389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1334.295226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1334.326540] LNet: 63014:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1334.350280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1334.376219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1334.418366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1334.465362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1334.514227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.554261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1334.608556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1334.927694] LNet: 63056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1334.945916] LNetError: 63056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.965057] LNet: Removed LNI 192.168.203.9@tcp [ 1335.358803] Key type .llcrypt unregistered [ 1335.361350] Key type ._llcrypt unregistered [ 1352.540412] Key type ._llcrypt registered [ 1352.545072] Key type .llcrypt registered [ 1354.190483] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:26:42 (1788931602) [ 1355.220341] Key type .llcrypt unregistered [ 1355.222628] Key type ._llcrypt unregistered [ 1362.425106] Key type ._llcrypt registered [ 1362.432316] Key type .llcrypt registered [ 1362.648347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1362.669324] alg: No test for adler32 (adler32-zlib) [ 1363.580627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1363.603170] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1372.025339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1382.445951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.505195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1382.533731] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1384.438193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_dropped -r 1 [ 1384.494344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e remote_dropped -r 1 [ 1384.535252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_dropped -r 1 [ 1384.598776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e remote_dropped -r 1 [ 1384.655787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e remote_dropped -r 1 [ 1384.710337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e remote_dropped -r 1 [ 1384.749642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e remote_dropped -r 1 [ 1384.794171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e remote_dropped -r 1 [ 1384.828942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1384.847372] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1384.851387] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.109@tcp [ 1385.125709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1385.148837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1385.170351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1385.200585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1385.234353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1385.272769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1385.307828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1385.342996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1385.377054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1385.400808] LNet: 64522:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1385.427457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1385.462156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1385.487821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1385.516826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1385.551062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1385.577223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1385.624969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1385.665491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1386.225671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_error -r 1 [ 1386.268862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e remote_error -r 1 [ 1386.306168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_error -r 1 [ 1386.347783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e remote_error -r 1 [ 1386.392143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e remote_error -r 1 [ 1386.426085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e remote_error -r 1 [ 1386.459341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e remote_error -r 1 [ 1386.495754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e remote_error -r 1 [ 1386.530704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1386.550862] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1386.557202] LNet: Skipped 2 previous similar messages [ 1386.777280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1386.818154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1386.899557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1386.941380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1386.980784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1387.052129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1387.104385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1387.127527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1387.169559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1387.197199] LNet: 64648:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1387.224062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1387.254573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1387.299050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1387.330900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1387.380612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1387.414914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1387.460692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1387.506747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1388.050728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_timeout -r 1 [ 1388.103919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e remote_timeout -r 1 [ 1388.153269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_timeout -r 1 [ 1388.203410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e remote_timeout -r 1 [ 1388.260792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e remote_timeout -r 1 [ 1388.335368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e remote_timeout -r 1 [ 1388.381763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e remote_timeout -r 1 [ 1388.447299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e remote_timeout -r 1 [ 1388.507718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1388.536732] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1388.803188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1388.845775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1388.899704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1388.947352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1389.017989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1389.079938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1389.118670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1389.182118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1389.228481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1389.258514] LNet: 64773:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1389.282618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1389.322316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1389.353328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1389.388610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1389.419586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1389.450541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1389.485980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1389.522595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1394.544326] LNet: 64851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.559879] LNetError: 64851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.566607] LNet: Removed LNI 192.168.203.9@tcp [ 1394.585642] LNet: Removed LNI 192.168.203.9@tcp1 [ 1395.067273] Key type .llcrypt unregistered [ 1395.072106] Key type ._llcrypt unregistered [ 1413.228612] Key type ._llcrypt registered [ 1413.231893] Key type .llcrypt registered [ 1414.719741] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:27:43 (1788931663) [ 1416.689306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1418.638956] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:27:47 (1788931667) [ 1419.494446] Key type .llcrypt unregistered [ 1419.499553] Key type ._llcrypt unregistered [ 1425.839074] Key type ._llcrypt registered [ 1425.845159] Key type .llcrypt registered [ 1426.092766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1426.107568] alg: No test for adler32 (adler32-zlib) [ 1427.038386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1427.087301] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1435.277872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1436.031112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e network_timeout -r 1 [ 1436.077578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e network_timeout -r 1 [ 1436.113660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1436.135641] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1436.326951] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1436.406852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1436.433655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1436.464669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1436.496394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1436.523384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1436.575667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1436.609847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1436.645041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1436.681991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1436.699429] LNet: 66396:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1436.720221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1436.786736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1436.825645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1436.890611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1436.927532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1436.963890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1437.248860] LNet: 66436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.260590] LNetError: 66436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.284371] LNet: Removed LNI 192.168.203.9@tcp [ 1437.760448] Key type .llcrypt unregistered [ 1437.766488] Key type ._llcrypt unregistered [ 1458.358835] Key type ._llcrypt registered [ 1458.364735] Key type .llcrypt registered [ 1458.788856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1458.813266] alg: No test for adler32 (adler32-zlib) [ 1459.794526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1459.852721] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1468.074663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1477.128980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.182226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1477.207577] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1479.194531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e network_timeout -r 1 [ 1479.242892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e network_timeout -r 1 [ 1479.283747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e network_timeout -r 1 [ 1479.329690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e network_timeout -r 1 [ 1479.359523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e network_timeout -r 1 [ 1479.399179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e network_timeout -r 1 [ 1479.442682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e network_timeout -r 1 [ 1479.490450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e network_timeout -r 1 [ 1479.535752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1479.558672] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1479.787791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1479.815920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1479.855478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1479.913535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1479.949134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1479.976867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1480.016828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1480.064686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1480.096833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1480.114570] LNet: 67313:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1480.134575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1480.162500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1480.187186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1480.220215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1480.277241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1480.322906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1480.353521] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1480.359925] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1482.400371] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1485.284037] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1485.737148] LNet: 67387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.746099] LNetError: 67387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.763330] LNet: Removed LNI 192.168.203.9@tcp [ 1485.781503] LNet: Removed LNI 192.168.203.9@tcp1 [ 1486.303541] Key type .llcrypt unregistered [ 1486.305315] Key type ._llcrypt unregistered [ 1503.723887] Key type ._llcrypt registered [ 1503.732591] Key type .llcrypt registered [ 1505.096933] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:29:13 (1788931753) [ 1505.508809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1505.524385] alg: No test for adler32 (adler32-zlib) [ 1506.439412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.471643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1506.488223] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1507.299618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1507.324410] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1508.216670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 1508.368352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1508.421650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1508.451319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1508.492711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1508.520581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1508.579396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.9@tcp [ 1508.625055] LNetError: 68484:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.9@tcp to recovery [ 1509.536317] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1509.536533] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1511.584332] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1512.230621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1515.681582] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1516.898468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1516.933086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1517.025679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.060777] LNet: 68601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.071521] LNetError: 68601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.083453] LNet: Removed LNI 192.168.203.9@tcp [ 1517.112430] LNet: Removed LNI 192.168.203.9@tcp1 [ 1517.141959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.229325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.257354] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1518.110389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.147776] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1519.072593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 1519.135706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.202699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1519.236113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1519.272877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1519.313712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1519.368589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1519.421377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.9@tcp [ 1519.443424] LNetError: 68874:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.9@tcp to recovery [ 1520.288441] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1523.071441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1526.432374] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1526.435716] LNet: Skipped 1 previous similar message [ 1531.088592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1531.141539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1531.185571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1538.199930] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:29:47 (1788931787) [ 1538.341658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.392700] LNet: 69587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.411069] LNetError: 69587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.422356] LNetError: 69587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1538.441595] LNet: Removed LNI 192.168.203.9@tcp [ 1538.485679] LNet: Removed LNI 192.168.203.9@tcp1 [ 1538.528788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.611925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1538.646259] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1539.481453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1539.509828] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1540.466073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 1540.576609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1540.625326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1540.662311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1540.689238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1540.737496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1540.795920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.9@tcp --health 0 [ 1540.924713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1541.664388] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.9@tcp [ 1541.664836] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1541.687671] LNet: Skipped 1 previous similar message [ 1556.587849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1556.799786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1556.840612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1556.863721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.9@tcp --health 500 [ 1556.962165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1562.574435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1562.657568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.695505] LNet: 70074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.710447] LNetError: 70074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.724112] LNetError: 70074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1562.738525] LNet: Removed LNI 192.168.203.9@tcp [ 1562.757922] LNet: Removed LNI 192.168.203.9@tcp1 [ 1562.812909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.875680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.918984] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1563.679721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1563.708694] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1564.509701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 1564.568992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1564.675275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1564.701232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1564.734462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1564.761918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1564.787104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1564.817521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.9@tcp --health 0 [ 1565.921232] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1565.929044] LNet: Skipped 3 previous similar messages [ 1566.154068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1575.256415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1575.310830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1575.369819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1582.154823] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:30:31 (1788931831) [ 1582.535756] LNet: 71109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.543659] LNetError: 71109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.553905] LNetError: 71109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1582.560784] LNet: Removed LNI 192.168.203.9@tcp [ 1582.570834] LNet: Removed LNI 192.168.203.9@tcp1 [ 1583.196435] Key type .llcrypt unregistered [ 1583.199445] Key type ._llcrypt unregistered [ 1588.887289] Key type ._llcrypt registered [ 1588.894602] Key type .llcrypt registered [ 1589.228878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1589.274315] alg: No test for adler32 (adler32-zlib) [ 1590.162873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1590.191955] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1599.070590] Lustre: DEBUG MARKER: Initial discovery [ 1599.252333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.109@tcp [ 1601.352595] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1601.481749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1601.534813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.109@tcp [ 1601.553194] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1601.573856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1601.605745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.109@tcp -r 1 -m GET -l 3 [ 1601.667600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.109@tcp -r 1 -m GET -e local_error [ 1601.734426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.109@tcp -r 1 -m PUT -l 6 [ 1602.529987] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1603.167022] Lustre: DEBUG MARKER: Force oleg309-client.virtnet to discover 192.168.203.109@tcp (in background) [ 1603.270031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.109@tcp [ 1605.143503] Lustre: DEBUG MARKER: Force oleg309-server to discover 192.168.203.9@tcp [ 1606.368262] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1607.703758] Lustre: DEBUG MARKER: Wait for 71860 [ 1609.510656] Lustre: DEBUG MARKER: Finished wait on 71860 [ 1609.562822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1609.614496] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1609.649849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1610.127793] LNet: 72317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.137427] LNetError: 72317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.161955] LNet: Removed LNI 192.168.203.9@tcp [ 1610.638819] Key type .llcrypt unregistered [ 1610.641275] Key type ._llcrypt unregistered [ 1628.218728] Key type ._llcrypt registered [ 1628.230766] Key type .llcrypt registered [ 1629.585721] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:31:18 (1788931878) [ 1630.143223] Key type .llcrypt unregistered [ 1630.145287] Key type ._llcrypt unregistered [ 1636.270811] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1636.432390] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1636.449289] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1636.462241] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1639.283297] Key type ._llcrypt registered [ 1639.291171] Key type .llcrypt registered [ 1639.569133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1639.597361] alg: No test for adler32 (adler32-zlib) [ 1640.582688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.653772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.675643] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1641.748972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1641.767654] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1650.988888] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:31:39 (1788931899) [ 1651.278662] LNet: 74389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1651.290803] LNetError: 74389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1651.306276] LNet: Removed LNI 10.1.2.3@tcp [ 1651.315648] LNet: Removed LNI 192.168.203.9@tcp [ 1652.201499] Key type .llcrypt unregistered [ 1652.210122] Key type ._llcrypt unregistered [ 1659.962422] Key type ._llcrypt registered [ 1659.965422] Key type .llcrypt registered [ 1660.266046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1660.292813] alg: No test for adler32 (adler32-zlib) [ 1661.264081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.348958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.365842] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1662.056964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1662.083581] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1663.266203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.9@tcp [ 1664.352202] LNetError: Unexpected error -22 connecting to 192.168.203.9@tcp at host 192.168.203.9:988 [ 1670.056967] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:31:59 (1788931919) [ 1670.440660] LNet: 75519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.449863] LNetError: 75519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1670.459315] LNet: Removed LNI 10.1.2.3@tcp [ 1670.470775] LNet: Removed LNI 192.168.203.9@tcp [ 1671.081155] Key type .llcrypt unregistered [ 1671.083506] Key type ._llcrypt unregistered [ 1677.193444] Key type ._llcrypt registered [ 1677.196415] Key type .llcrypt registered [ 1677.531333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1677.549748] alg: No test for adler32 (adler32-zlib) [ 1678.564715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.656958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1678.688823] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1679.554717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1679.590782] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 1680.627578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.9@tcp1 --nid 192.168.203.9@tcp2 [ 1685.207870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp1 192.168.203.9@tcp2 [ 1685.227576] LNetError: 76351:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.9@tcp1 to 192.168.203.9@tcp2 - Route table may be misconfigured [ 1685.254042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp2 192.168.203.9@tcp1 [ 1691.199246] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:32:20 (1788931940) [ 1691.315352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.382955] LNet: 76916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.397977] LNetError: 76916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.422271] LNet: Removed LNI 192.168.203.9@tcp1 [ 1692.455044] LNet: Removed LNI 192.168.203.9@tcp2 [ 1692.529322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.640614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1692.717501] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1693.654713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1693.680919] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1694.525357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 1694.571280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.9@tcp [ 1694.616929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp1 192.168.203.9@tcp1 [ 1694.663461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.9@tcp -d 192.168.203.9@tcp -e network_timeout [ 1694.727933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -e network_timeout [ 1694.770828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -e network_timeout [ 1694.813140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -e network_timeout [ 1694.848894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1694.875068] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1694.900104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1695.006116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1695.123641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1695.649800] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1701.099436] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:32:30 (1788931950) [ 1701.340693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.359268] LNet: 77785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.364243] LNetError: 77785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.385378] LNetError: 77785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1701.394271] LNet: Removed LNI 192.168.203.9@tcp [ 1702.439838] LNet: Removed LNI 192.168.203.9@tcp1 [ 1702.471349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.568081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1702.850770] LNet: 77822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.258184] Key type .llcrypt unregistered [ 1703.267039] Key type ._llcrypt unregistered [ 1714.461490] Key type ._llcrypt registered [ 1714.464220] Key type .llcrypt registered [ 1715.714154] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:32:44 (1788931964) [ 1716.226505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.242393] alg: No test for adler32 (adler32-zlib) [ 1717.160336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.473377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1720.178808] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:32:49 (1788931969) [ 1720.292882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.323445] LNet: 78914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.378946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.464436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1720.508303] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1721.426879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1721.451717] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1722.426156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1722.482859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp1 [ 1722.527513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp1 [ 1728.501134] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:32:57 (1788931977) [ 1730.273409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg309-server [ 1731.880304] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:33:01 (1788931981) [ 1733.604318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg309-server [ 1735.395409] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:33:04 (1788931984) [ 1737.496276] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg309-server [ 1739.051242] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:33:08 (1788931988) [ 1740.969530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg309-server [ 1742.726838] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:33:11 (1788931991) [ 1744.545877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg309-server [ 1746.137133] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:33:15 (1788931995) [ 1747.916577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg309-server [ 1749.576937] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:33:18 (1788931998) [ 1751.324701] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg309-server [ 1753.203988] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:33:22 (1788932002) [ 1755.428690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg309-server [ 1757.219942] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:33:26 (1788932006) [ 1759.176882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg309-server [ 1761.177846] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:33:30 (1788932010) [ 1761.442050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.467884] LNet: 82801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.476789] LNetError: 82801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.528389] LNet: Removed LNI 192.168.203.9@tcp [ 1762.530515] LNetError: 82801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.549368] LNet: Removed LNI 192.168.203.9@tcp1 [ 1762.596255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.675305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1762.698864] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1763.502447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1763.629312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1763.897752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.941323] LNet: 82949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.961498] LNetError: 82949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.982380] LNet: Removed LNI 192.168.203.9@tcp [ 1764.032842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.166165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1764.191391] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1765.222793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1765.556268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.630485] LNet: Removed LNI 192.168.203.9@tcp [ 1766.693573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.763283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1766.792300] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1767.649787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1767.734453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1767.988994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.007449] LNet: 83241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.012628] LNet: 83241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1768.019378] LNetError: 83241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.027651] LNetError: 83241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1768.053885] LNet: Removed LNI 192.168.203.9@tcp [ 1768.133562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.209688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1768.242926] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1769.099063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1769.324409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.413709] LNet: Removed LNI 192.168.203.9@tcp [ 1770.442957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.518964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1770.559486] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1771.492801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1771.574398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1771.822363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.903983] LNet: Removed LNI 192.168.203.9@tcp [ 1772.937868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.015855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1773.046711] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1773.863361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1774.072302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.089723] LNet: 83679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.094669] LNet: 83679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1774.104251] LNetError: 83679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.110440] LNetError: 83679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1775.142626] LNet: Removed LNI 192.168.203.9@tcp [ 1775.169482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.236761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1775.272494] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1775.976922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1776.068292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1776.317486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.383734] LNet: Removed LNI 192.168.203.9@tcp [ 1777.405744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.487589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1777.527842] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1778.306601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1778.549403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.623230] LNet: Removed LNI 192.168.203.9@tcp [ 1779.661216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.714822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1779.760845] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1780.408818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1780.493256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1780.742850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.812993] LNet: Removed LNI 192.168.203.9@tcp [ 1781.875843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.957595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1781.991895] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1782.853700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1783.113498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.128574] LNet: 84264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.133490] LNet: 84264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1783.138994] LNetError: 84264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.146450] LNetError: 84264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1783.160552] LNet: Removed LNI 192.168.203.9@tcp [ 1783.179236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.226293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1783.247347] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1783.880336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1783.961168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1785.410500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.470735] LNet: Removed LNI 192.168.203.9@tcp [ 1786.507579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.588981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1786.625817] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1787.543898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1787.838526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.902107] LNet: Removed LNI 192.168.203.9@tcp [ 1788.947201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.011680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1789.043478] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1789.844615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1789.920599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.954706] LNet: Removed LNI 192.168.203.9@tcp [ 1789.988588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.059676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1790.095819] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1790.845244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1790.906646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1790.934595] LNet: Removed LNI 192.168.203.9@tcp [ 1790.968849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.015372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1791.109033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.134515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.169248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1791.225271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.247595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.283137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1791.341720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.365390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.415391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1791.442712] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1792.285763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.318943] LNet: Removed LNI 192.168.203.9@tcp [ 1792.337438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.471492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1799.469359] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:34:08 (1788932048) [ 1799.736944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.754066] LNet: 85587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.760332] LNet: 85587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1799.777448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.875546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1799.905691] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/180] [ 1801.008066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.045652] LNetError: 85725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.052022] LNetError: 85725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1801.062237] LNet: Removed LNI 192.168.203.9@tcp231 [ 1801.093179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.174326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1801.231694] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/180] [ 1801.240304] LNet: 85733:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1801.600811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1801.627655] LNet: Removed LNI 192.168.203.9@tcp231 [ 1801.640030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1801.669052] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/180] [ 1802.645692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.669220] LNet: Removed LNI 192.168.203.9@tcp231 [ 1802.684692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.740285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1802.760965] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/0] [ 1802.765907] LNet: 85885:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1803.124922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1803.170965] LNet: Removed LNI 192.168.203.9@tcp231 [ 1803.194838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1803.233125] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/0] [ 1804.283119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.320739] LNet: Removed LNI 192.168.203.9@tcp231 [ 1804.379171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.494236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1804.556109] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/60] [ 1804.573061] LNet: 86036:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1805.141825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1805.167818] LNet: Removed LNI 192.168.203.9@tcp231 [ 1805.178611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1805.193763] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/60] [ 1806.368439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.417554] LNet: Removed LNI 192.168.203.9@tcp231 [ 1806.452219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.577779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1806.607270] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/180] [ 1806.613287] LNet: 86188:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1807.200659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1807.274285] LNet: Removed LNI 192.168.203.9@tcp231 [ 1807.305247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1807.341653] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/180] [ 1817.425814] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:34:25 (1788932065) [ 1817.562915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.643114] LNet: Removed LNI 192.168.203.9@tcp231 [ 1817.733896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.938564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1818.005331] LNet: Added LNI 192.168.203.9@tcp232 [8/256/0/180] [ 1819.252619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1819.314367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1819.361348] LNet: Removed LNI 192.168.203.9@tcp232 [ 1819.399958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1819.441767] LNet: Added LNI 192.168.203.9@tcp232 [8/256/0/180] [ 1820.661685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1820.736325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1820.777045] LNet: Removed LNI 192.168.203.9@tcp232 [ 1820.816978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1820.912535] LNet: Added LNI 192.168.203.9@tcp232 [8/256/0/180] [ 1822.154774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1822.224179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1822.274804] LNet: Removed LNI 192.168.203.9@tcp232 [ 1822.303109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1822.335361] LNet: Added LNI 192.168.203.9@tcp232 [8/256/0/180] [ 1823.322335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1823.364330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1823.409047] LNet: Removed LNI 192.168.203.9@tcp232 [ 1823.430231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1829.687235] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:34:38 (1788932078) [ 1831.875978] Key type .llcrypt unregistered [ 1831.878361] Key type ._llcrypt unregistered [ 1839.678306] Key type ._llcrypt registered [ 1839.689516] Key type .llcrypt registered [ 1839.926905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1839.938747] alg: No test for adler32 (adler32-zlib) [ 1840.778031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1840.795133] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1848.812699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1861.578823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.623968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1861.684180] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1864.104699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_interrupt -r 2 [ 1864.158364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_interrupt -r 2 [ 1864.201136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_interrupt -r 2 [ 1864.285519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_interrupt -r 2 [ 1864.386567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_interrupt -r 2 [ 1864.456873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_interrupt -r 2 [ 1864.516869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_interrupt -r 2 [ 1864.571856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_interrupt -r 2 [ 1864.638538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1864.677507] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1864.692743] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1865.046276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1865.074608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1865.111906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1865.159059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1865.197230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1865.225840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1865.254786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1865.294668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1865.346654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1865.370159] LNet: 88690:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1865.386521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1865.430532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1865.469611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1865.499104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1865.552950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1865.604683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1866.259971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_dropped -r 2 [ 1866.294586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_dropped -r 2 [ 1866.348720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_dropped -r 2 [ 1866.394835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_dropped -r 2 [ 1866.467538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_dropped -r 2 [ 1866.503271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_dropped -r 2 [ 1866.573588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_dropped -r 2 [ 1866.623897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_dropped -r 2 [ 1866.706091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1866.742836] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1866.988405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1867.020686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1867.056461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1867.081274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1867.107864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1867.143131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1867.175481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1867.215944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1867.251665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1867.269205] LNet: 88812:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1867.284075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1867.315107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1867.349109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1867.388830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1867.413802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1867.464409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1868.245572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_aborted -r 2 [ 1868.313861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_aborted -r 2 [ 1868.376053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_aborted -r 2 [ 1868.453453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_aborted -r 2 [ 1868.485540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_aborted -r 2 [ 1868.530056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_aborted -r 2 [ 1868.601319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_aborted -r 2 [ 1868.648554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_aborted -r 2 [ 1868.694493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1868.710590] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1869.116694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1869.147140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1869.232582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1869.312810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1869.368112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1869.427292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1869.461969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1869.520405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1869.580677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1869.612398] LNet: 88934:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1869.640451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1869.683193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1869.717441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1869.776455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1869.836065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1869.911253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1870.880568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_no_route -r 2 [ 1870.932790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_no_route -r 2 [ 1870.995450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_no_route -r 2 [ 1871.045484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_no_route -r 2 [ 1871.106442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_no_route -r 2 [ 1871.189741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_no_route -r 2 [ 1871.287542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_no_route -r 2 [ 1871.381852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_no_route -r 2 [ 1871.478427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1871.535552] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1872.027277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1872.089218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1872.167700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1872.237830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1872.287604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1872.343683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1872.379456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1872.427917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1872.484967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1872.501983] LNet: 89057:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1872.541355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1872.573821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1872.593551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1872.612730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1872.672963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1872.721603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1873.288170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_timeout -r 2 [ 1873.333532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_timeout -r 2 [ 1873.381056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_timeout -r 2 [ 1873.435429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_timeout -r 2 [ 1873.469150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_timeout -r 2 [ 1873.492604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_timeout -r 2 [ 1873.514379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_timeout -r 2 [ 1873.553240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_timeout -r 2 [ 1873.603726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1873.778698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1873.800532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1873.820573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1873.839713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1873.855034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1873.871530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1873.903748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1873.923146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1873.938280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1873.966344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1873.993417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1874.015249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1874.038762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1874.068106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1874.100280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1874.695865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_dropped -r 2 [ 1874.724344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e remote_dropped -r 2 [ 1874.748642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_dropped -r 2 [ 1874.784661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e remote_dropped -r 2 [ 1874.821371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e remote_dropped -r 2 [ 1874.865457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e remote_dropped -r 2 [ 1874.900427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e remote_dropped -r 2 [ 1874.938912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e remote_dropped -r 2 [ 1874.991442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1875.019673] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.109@tcp [ 1875.300737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1875.329387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1875.367787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1875.424931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1875.518818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1875.576023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1875.642773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1875.717258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1875.779878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1875.812502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1875.878785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1875.954573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1875.990783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1876.021422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1876.084621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1883.309323] LNet: 89375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.332942] LNetError: 89375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.342354] LNet: Removed LNI 192.168.203.9@tcp [ 1883.361933] LNet: Removed LNI 192.168.203.9@tcp1 [ 1884.167854] Key type .llcrypt unregistered [ 1884.179854] Key type ._llcrypt unregistered [ 1901.668629] Key type ._llcrypt registered [ 1901.672459] Key type .llcrypt registered [ 1903.108767] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:35:51 (1788932151) [ 1903.653139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.691806] alg: No test for adler32 (adler32-zlib) [ 1904.622568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.673831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1904.708587] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1905.634438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1905.671804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.9@tcp [ 1905.696322] LNetError: 90351:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.9@tcp to recovery [ 1905.707166] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1905.806125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.9@tcp [ 1905.951846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1912.395434] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:36:01 (1788932161) [ 1912.508445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.543985] LNet: 90934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.554367] LNetError: 90934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.569483] LNet: Removed LNI 192.168.203.9@tcp [ 1912.593259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.647419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1912.673461] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1913.576605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1913.610713] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1914.551445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.9@tcp [ 1914.579711] LNetError: 91207:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.9@tcp to recovery [ 1914.660587] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1915.957891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.9@tcp [ 1923.896431] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:36:12 (1788932172) [ 1925.872744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg309-server [ 1927.558668] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:36:16 (1788932176) [ 1929.277644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg309-server [ 1931.093376] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:36:20 (1788932180) [ 1933.358524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg309-server [ 1935.170411] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:36:24 (1788932184) [ 1935.300731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1935.346866] LNet: 92824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.356839] LNetError: 92824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.366675] LNet: Removed LNI 192.168.203.9@tcp [ 1935.405769] LNet: Removed LNI 192.168.203.9@tcp1 [ 1935.465463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.537479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1935.561301] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1936.347456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1936.419215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1936.500606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1936.670606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1936.720296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1936.784415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1945.183571] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:36:33 (1788932193) [ 1945.431401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1945.462143] LNet: 93561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.471184] LNetError: 93561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.480447] LNetError: 93561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1945.498286] LNet: Removed LNI 192.168.203.9@tcp [ 1945.530926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.606566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1945.629876] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1953.900724] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:36:42 (1788932202) [ 1954.440182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1954.469943] LNet: 94265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.501377] LNetError: 94265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.527727] LNet: Removed LNI 192.168.203.9@tcp [ 1954.591960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.665664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1954.707531] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1962.425626] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:36:51 (1788932211) [ 1964.268763] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1966.196971] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:36:55 (1788932215) [ 1966.856832] LNet: 95300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.871217] LNetError: 95300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.885569] LNet: Removed LNI 192.168.203.9@tcp [ 1967.683918] Key type .llcrypt unregistered [ 1967.688582] Key type ._llcrypt unregistered [ 1974.167184] Key type ._llcrypt registered [ 1974.168910] Key type .llcrypt registered [ 1974.578103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1974.592352] alg: No test for adler32 (adler32-zlib) [ 1975.579693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1975.637791] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1983.377662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1989.090446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.109@tcp [ 1990.491971] LNet: 95715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.506682] LNetError: 95715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.544362] LNet: Removed LNI 192.168.203.9@tcp [ 1991.039716] Key type .llcrypt unregistered [ 1991.046837] Key type ._llcrypt unregistered [ 2003.465341] Key type ._llcrypt registered [ 2003.468764] Key type .llcrypt registered [ 2004.698954] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2006.623186] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2008.617870] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:37:37 (1788932257) [ 2010.370076] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2012.154538] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:37:41 (1788932261) [ 2014.142595] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg309-server [ 2016.142480] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:37:45 (1788932265) [ 2018.068470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg309-server [ 2019.668411] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:37:48 (1788932268) [ 2021.983508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2022.165938] alg: No test for adler32 (adler32-zlib) [ 2023.311972] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 2023.535684] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2025.216652] Key type lgssc registered [ 2026.540371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2050.054717] Key type lgssc unregistered [ 2050.369877] LNet: 98887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.376614] LNetError: 98887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.394626] LNet: Removed LNI 192.168.203.9@tcp [ 2051.212681] Key type .llcrypt unregistered [ 2051.214383] Key type ._llcrypt unregistered [ 2067.585761] Key type ._llcrypt registered [ 2067.587603] Key type .llcrypt registered [ 2069.090821] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:38:37 (1788932317) [ 2069.565365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2069.604710] alg: No test for adler32 (adler32-zlib) [ 2070.692716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.833706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2070.920409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2071.029101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2071.113890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2079.266776] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:38:48 (1788932328) [ 2079.386088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.437508] LNet: 100293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.483955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.610255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2079.678041] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2080.942293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2080.967214] LNet: 100443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.981588] LNetError: 100443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2081.021846] LNet: Removed LNI 192.168.203.9@tcp [ 2081.039816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2081.125525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2081.158581] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2081.797505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2081.823064] LNetError: 100472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2081.851540] LNet: Removed LNI 192.168.203.9@tcp [ 2081.887637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2081.962295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2081.994465] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2089.244164] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:38:58 (1788932338) [ 2089.698859] LNet: 101062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.706124] LNet: 101062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2089.726412] LNetError: 101062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.760570] LNet: Removed LNI 192.168.203.9@tcp [ 2090.429878] Key type .llcrypt unregistered [ 2090.432775] Key type ._llcrypt unregistered [ 2097.532509] Key type ._llcrypt registered [ 2097.538690] Key type .llcrypt registered [ 2097.883968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2097.916416] alg: No test for adler32 (adler32-zlib) [ 2098.925995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2099.039098] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2099.044060] LNet: *** cfs_fail_loc=e003, val=100*** [ 2100.039575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2100.062230] LNet: *** cfs_fail_loc=e003, val=100*** [ 2100.065111] LNet: Skipped 2 previous similar messages [ 2100.079317] Ping reply received with mapping: [ 2100.081203] found metadata reply NI[192.168.203.9@tcp] [ 2100.090775] 09 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2100.100079] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2100.101979] 5a 5a 5a 5a 5a 5a 5a 5a [ 2100.303177] LNet: 101457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.319603] LNetError: 101457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.348331] LNet: Removed LNI 192.168.203.9@tcp [ 2101.281750] Key type .llcrypt unregistered [ 2101.286258] Key type ._llcrypt unregistered [ 2113.777219] Key type ._llcrypt registered [ 2113.780640] Key type .llcrypt registered [ 2115.096842] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:39:24 (1788932364) [ 2115.674306] Key type .llcrypt unregistered [ 2115.676492] Key type ._llcrypt unregistered [ 2122.579537] Key type ._llcrypt registered [ 2122.581055] Key type .llcrypt registered [ 2122.864509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2122.904347] alg: No test for adler32 (adler32-zlib) [ 2123.926871] LNetError: 102496:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2124.321349] Key type .llcrypt unregistered [ 2124.324833] Key type ._llcrypt unregistered [ 2129.249645] Key type ._llcrypt registered [ 2129.251615] Key type .llcrypt registered [ 2130.737505] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:39:39 (1788932379) [ 2131.239310] Key type .llcrypt unregistered [ 2131.241819] Key type ._llcrypt unregistered [ 2137.752956] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2137.775023] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2140.485284] Key type ._llcrypt registered [ 2140.490942] Key type .llcrypt registered [ 2140.781179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2140.817627] alg: No test for adler32 (adler32-zlib) [ 2141.851673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.937191] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2151.341204] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:40:00 (1788932400) [ 2151.459508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.492271] LNet: 104264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.508987] LNetError: 104264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2151.530290] LNet: Removed LNI 10.1.2.3@tcp [ 2151.564811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.812794] Key type .llcrypt unregistered [ 2152.816089] Key type ._llcrypt unregistered [ 2159.499641] Key type ._llcrypt registered [ 2159.505991] Key type .llcrypt registered [ 2159.716124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2159.733745] alg: No test for adler32 (adler32-zlib) [ 2160.711043] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2160.847992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2161.084678] LNet: 104556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.104742] LNetError: 104556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.131982] LNet: Removed LNI 192.168.203.9@tcp [ 2161.831947] Key type .llcrypt unregistered [ 2161.838747] Key type ._llcrypt unregistered [ 2174.610172] Key type ._llcrypt registered [ 2174.612129] Key type .llcrypt registered [ 2175.979252] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:40:24 (1788932424) [ 2176.822522] Key type .llcrypt unregistered [ 2176.825467] Key type ._llcrypt unregistered [ 2183.409791] Key type ._llcrypt registered [ 2183.411940] Key type .llcrypt registered [ 2183.741447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2183.763941] alg: No test for adler32 (adler32-zlib) [ 2192.081100] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:40:40 (1788932440) [ 2192.378064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2192.442591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2192.540924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2192.580660] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2193.199432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2193.232971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2193.256204] LNet: Removed LNI 192.168.203.9@tcp [ 2193.436256] LNet: 106296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.840336] Key type .llcrypt unregistered [ 2193.843920] Key type ._llcrypt unregistered [ 2205.548857] Key type ._llcrypt registered [ 2205.552848] Key type .llcrypt registered [ 2206.945652] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:40:55 (1788932455) [ 2207.375453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2207.410343] alg: No test for adler32 (adler32-zlib) [ 2208.402328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.467931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2208.519644] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2209.536898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2209.604745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.9@tcp [ 2209.661682] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2209.670977] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2209.677524] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2209, 0, 0 [ 2209.686272] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2209.691513] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2209.695347] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2209.698358] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2209, 0, 0 [ 2209.705577] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2209.712287] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2209.716529] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2209.719953] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2209, 0, 2259 [ 2209.724612] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2209.727140] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2209.730445] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2209.733825] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2209, 0, 2259 [ 2209.748964] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2217.976765] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:41:06 (1788932466) [ 2218.560357] LNet: 107835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.582874] LNetError: 107835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.615320] LNet: Removed LNI 192.168.203.9@tcp [ 2219.921745] Key type .llcrypt unregistered [ 2219.923774] Key type ._llcrypt unregistered [ 2229.335300] Key type ._llcrypt registered [ 2229.338586] Key type .llcrypt registered [ 2229.484693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2229.502298] alg: No test for adler32 (adler32-zlib) [ 2230.406903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2230.427412] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2239.444106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 2248.983664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2249.038696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2249.068661] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 2249.993444] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2250.143489] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2250.149251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2250.155170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2252.473801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2252.525512] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2263.744364] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:41:52 (1788932512) [ 2263.922830] LNet: 109341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.934828] LNetError: 109341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.944545] LNet: Removed LNI 192.168.203.9@tcp [ 2264.992279] LNet: Removed LNI 192.168.203.9@tcp1 [ 2265.004272] LNetError: 109341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2265.018356] LNetError: 109341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2265.038075] LNet: Removed LNI 10.1.2.3@tcp99 [ 2265.771494] Key type .llcrypt unregistered [ 2265.773667] Key type ._llcrypt unregistered [ 2271.565285] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2271.571936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2274.145689] Key type ._llcrypt registered [ 2274.150977] Key type .llcrypt registered [ 2274.314838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2274.325383] alg: No test for adler32 (adler32-zlib) [ 2275.259226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.322789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2275.349177] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2276.000292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2276.032274] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2278.146188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2278.190478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2278.232801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2278.252076] LNetError: 110087:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2278.294884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2278.340742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2278.825438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2278.918686] LNet: 110107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.928754] LNetError: 110107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2278.945096] LNet: Removed LNI 10.1.2.3@tcp [ 2279.976995] LNet: Removed LNI 192.168.203.9@tcp [ 2280.795413] Key type .llcrypt unregistered [ 2280.798061] Key type ._llcrypt unregistered [ 2293.948854] Key type ._llcrypt registered [ 2293.950762] Key type .llcrypt registered [ 2295.241961] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:42:24 (1788932544) [ 2295.762394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2295.792376] alg: No test for adler32 (adler32-zlib) [ 2296.780118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.859523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2296.882940] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2297.502296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2297.600807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg309-client@tcp [ 2303.541421] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:42:32 (1788932552) [ 2303.677233] LNet: 111636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.684731] LNetError: 111636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.705872] LNet: Removed LNI 192.168.203.9@tcp [ 2304.297570] Key type .llcrypt unregistered [ 2304.304784] Key type ._llcrypt unregistered [ 2310.749368] Key type ._llcrypt registered [ 2310.752553] Key type .llcrypt registered [ 2310.903893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.913418] alg: No test for adler32 (adler32-zlib) [ 2311.811514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2311.849285] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2311.878612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2319.251952] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:42:48 (1788932568) [ 2320.977735] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2322.673649] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:42:51 (1788932571) [ 2324.178922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2325.837723] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:42:54 (1788932574) [ 2326.249744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.281992] LNet: 113057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2326.286375] LNetError: 113057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.301262] LNet: Removed LNI 192.168.203.9@tcp [ 2326.318832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.389659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.413299] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2327.204788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2327.240300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2334.382948] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:43:03 (1788932583) [ 2334.579990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.601032] LNet: 113767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.605425] LNetError: 113767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.622851] LNet: Removed LNI 192.168.203.9@tcp [ 2334.656962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.711945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2334.739363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2335.362324] Key type .llcrypt unregistered [ 2335.366715] Key type ._llcrypt unregistered [ 2347.322770] Key type ._llcrypt registered [ 2347.326577] Key type .llcrypt registered [ 2348.530385] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:43:17 (1788932597) [ 2348.846746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2348.857499] alg: No test for adler32 (adler32-zlib) [ 2349.784144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.861265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.886986] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2350.724773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2350.781971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2358.053824] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:43:26 (1788932606) [ 2358.175122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.197916] LNet: 115315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2358.224987] LNetError: 115315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.254373] LNet: Removed LNI 192.168.203.9@tcp [ 2358.317962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.370717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2358.408524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2365.191118] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:43:34 (1788932614) [ 2365.603695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.624533] LNet: 115901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.642680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.714111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2366.112482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2366.150356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2366.212330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2366.687574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2366.707848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2366.760653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2366.781989] LNet: 115934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.805644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.867877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2367.247205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2367.281625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2367.339481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2367.911338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2367.943740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2373.838034] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:43:43 (1788932623) [ 2373.982237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.022723] LNet: 116535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.046542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2374.094234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2374.125648] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2374.779885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2374.808255] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2375.496561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2375.554628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2375.739813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2375.758102] LNetError: 116814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2376.800167] LNet: Removed LNI 192.168.203.9@tcp [ 2376.806226] LNetError: 116814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2376.835988] LNet: Removed LNI 192.168.203.9@tcp2 [ 2376.854529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2376.899462] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2376.903672] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2376.907856] LNet: 116820:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2377.341476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2377.529832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2378.592246] LNet: Removed LNI 192.168.203.9@tcp [ 2378.604974] LNet: Removed LNI 192.168.203.9@tcp2 [ 2378.651345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.715851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2378.745599] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2379.269887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2379.295587] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2380.118259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2380.185386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2380.238909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2380.429801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.451785] LNet: 117142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.456871] LNet: 117142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2380.462549] LNetError: 117142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2380.473933] LNetError: 117142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2380.485401] LNet: Removed LNI 192.168.203.9@tcp [ 2381.542732] LNet: Removed LNI 192.168.203.9@tcp2 [ 2381.581516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2381.669887] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2381.676120] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2381.682845] LNet: 117148:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2382.205496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2382.475658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2382.495842] LNet: Removed LNI 192.168.203.9@tcp [ 2383.534606] LNet: Removed LNI 192.168.203.9@tcp2 [ 2383.571620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.650589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2383.685631] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2384.442592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2384.463372] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2385.200696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2385.277160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2385.505876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2385.520881] LNetError: 117465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2385.525480] LNetError: 117465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2385.529611] LNet: Removed LNI 192.168.203.9@tcp [ 2385.546639] LNet: Removed LNI 192.168.203.9@tcp2 [ 2385.569946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2385.635995] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2385.641647] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2385.646316] LNet: 117471:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2386.425885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2386.654422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2387.744177] LNet: Removed LNI 192.168.203.9@tcp [ 2387.754829] LNet: Removed LNI 192.168.203.9@tcp2 [ 2387.807747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.876975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2387.904815] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2388.657175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2388.681027] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2389.348988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2389.419688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2389.501081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2389.714267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2389.733450] LNet: 117792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2389.738368] LNet: 117792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2390.752156] LNet: Removed LNI 192.168.203.9@tcp [ 2391.782586] LNet: Removed LNI 192.168.203.9@tcp2 [ 2391.824392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2391.914650] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2391.926812] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2391.940384] LNet: 117799:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2392.592377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2392.838467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.858226] LNet: Removed LNI 192.168.203.9@tcp [ 2393.895454] LNet: Removed LNI 192.168.203.9@tcp2 [ 2393.934136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.016393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2394.032958] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2394.792496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2394.818618] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2395.699594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2395.735478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2395.775979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2396.121961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2396.148664] LNetError: 118133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.157840] LNetError: 118133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2396.163681] LNet: Removed LNI 192.168.203.9@tcp [ 2396.176531] LNet: Removed LNI 192.168.203.9@tcp2 [ 2396.200740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.261727] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2396.267125] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2396.271130] LNet: 118139:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2396.645961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2396.970798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2398.048199] LNet: Removed LNI 192.168.203.9@tcp [ 2398.055697] LNet: Removed LNI 192.168.203.9@tcp2 [ 2398.097285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.187401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2398.230559] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2398.961370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2398.981767] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2399.775740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2399.821758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2399.861417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2399.930989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2400.275847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.313455] LNet: Removed LNI 192.168.203.9@tcp [ 2402.347167] LNet: Removed LNI 192.168.203.9@tcp2 [ 2402.385881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.474591] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2402.489256] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2402.964846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2409.436435] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2410.913545] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:44:20 (1788932660) [ 2411.105753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2411.126791] LNet: 119259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2411.135363] LNet: 119259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2411.140758] LNet: Removed LNI 192.168.203.9@tcp [ 2412.199219] LNet: Removed LNI 192.168.203.9@tcp2 [ 2412.228601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2412.566847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2412.592911] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2413.224580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2413.242282] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2414.074677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2414.090678] LNetError: 119532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2414.095927] LNetError: 119532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2414.101808] LNet: Removed LNI 10.1.2.3@tcp [ 2414.112144] LNet: Removed LNI 192.168.203.9@tcp [ 2414.141876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2414.197425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2414.217172] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2414.221613] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2414.618601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2414.647833] LNet: Removed LNI 192.168.203.9@tcp [ 2414.673839] LNet: Removed LNI 10.1.2.3@tcp [ 2421.067386] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:44:30 (1788932670) [ 2421.294393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2421.360744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.420121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2421.445631] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2422.216141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2422.255158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2422.310211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2422.359181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2422.389545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2422.433759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2422.477810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2422.503980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2422.526155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2422.564203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2422.627821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2422.662502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2422.865653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2429.801738] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 01:44:38 (1788932678) [ 2430.015426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2431.142920] LNet: Removed LNI 192.168.203.9@tcp [ 2431.190507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.319260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2431.359809] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2432.041654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2432.065984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2432.107550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.161585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.189083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.229108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.268191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.302936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.345273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.383322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.434657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.467817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.599901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2432.629632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.675838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.743877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.787392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.819885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.871400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.899269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2432.959281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2433.003545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2433.060227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2439.351533] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:44:48 (1788932688) [ 2439.464626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2440.557822] LNet: Removed LNI 192.168.203.9@tcp [ 2440.576118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.759929] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2440.862249] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2440.866490] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2440.871699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2443.152098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2443.209078] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2443.799540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2443.825794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2445.831278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2445.885202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2445.979819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.9@tcp -m PUT -e local_timeout -r 1 [ 2446.060752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2446.129727] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2446.138976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2446.514148] LNetError: 121694:0:(lib-move.c:4794:lnet_parse()) 192.168.203.9@tcp, src 192.168.203.9@tcp: Dropping PUT (error -108 looking up sender) [ 2447.528173] LNet: 122154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.541614] LNet: 122154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2447.547953] LNetError: 122154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2447.557339] LNetError: 122154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2447.564606] LNet: Removed LNI 10.1.2.3@tcp [ 2448.619477] LNet: Removed LNI 192.168.203.9@tcp [ 2449.202798] Key type .llcrypt unregistered [ 2449.204155] Key type ._llcrypt unregistered [ 2464.705364] Key type ._llcrypt registered [ 2464.713297] Key type .llcrypt registered [ 2466.885426] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:45:14 (1788932714) [ 2468.163277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2468.209742] alg: No test for adler32 (adler32-zlib) [ 2469.131392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2469.663863] LNet: 123004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2470.728933] Key type .llcrypt unregistered [ 2470.730727] Key type ._llcrypt unregistered [ 2478.289607] Key type ._llcrypt registered [ 2478.293564] Key type .llcrypt registered [ 2478.577638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2478.603301] alg: No test for adler32 (adler32-zlib) [ 2479.669796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2479.701675] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2489.076222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 2489.275310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2489.871715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.109@tcp [ 2489.957184] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.109@tcp [ 2490.217986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2490.238114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2490.267499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2490.299740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2490.361705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2490.404440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2490.448679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2490.472524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2490.501951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2490.518746] LNet: 123453:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2490.541858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2490.591073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2490.621721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2490.652459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2490.712375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2490.773174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2491.445635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2491.781265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2491.842991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2491.922148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2491.994831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2492.040377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2492.082973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2492.163588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2492.208952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2492.268069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2492.297910] LNet: 123573:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2492.316172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2492.353693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2492.391979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2492.474710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2492.521464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2492.560664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2493.545101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.109@tcp [ 2493.947610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2493.998467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2494.046385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2494.100743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2494.183302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2494.302428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2494.348813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2494.390771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2494.417633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2494.431887] LNet: 123692:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2494.469900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2494.514520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2494.580902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2494.624495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2494.679974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2494.737968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2495.232292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2495.477742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2495.501458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2495.525491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2495.554874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2495.577228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2495.596243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2495.618771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2495.654556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2495.674385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2495.694369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2495.722814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2495.739843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2495.781234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2495.825666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2495.855291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2496.163356] LNet: 123857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.169525] LNetError: 123857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.187745] LNet: Removed LNI 192.168.203.9@tcp [ 2496.560873] Key type .llcrypt unregistered [ 2496.566369] Key type ._llcrypt unregistered [ 2512.723691] Key type ._llcrypt registered [ 2512.725633] Key type .llcrypt registered [ 2513.885766] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:46:02 (1788932762) [ 2514.261166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2514.282409] alg: No test for adler32 (adler32-zlib) [ 2515.252801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2515.695289] LNet: 124741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2516.357415] Key type .llcrypt unregistered [ 2516.361939] Key type ._llcrypt unregistered [ 2523.385081] Key type ._llcrypt registered [ 2523.387466] Key type .llcrypt registered [ 2523.641488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2523.661439] alg: No test for adler32 (adler32-zlib) [ 2524.599234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2524.642115] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2532.461366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 2541.121088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2541.158405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2541.179525] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 2542.136776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2542.702102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.109@tcp [ 2542.912861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2542.939702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2542.965652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2542.996316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2543.031170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2543.067967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2543.082509] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.109@tcp [ 2543.115062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2543.160515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2543.221724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2543.261039] LNet: 125392:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2543.297757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2543.352392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2543.430209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2543.482872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2543.557548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2543.622822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2544.139806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2544.324456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2544.356469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2544.385975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2544.421543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2544.443789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2544.467266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2544.493129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2544.532919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2544.566082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2544.576841] LNet: 125511:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2544.596934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2544.648763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2544.685550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2544.710904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2544.753208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2544.803242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2545.279719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.109@tcp [ 2545.439834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2545.493763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2545.521441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2545.559708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2545.588247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2545.639873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2545.688254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2545.738257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2545.793612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2545.824462] LNet: 125630:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2545.849675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2545.886121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2545.916579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2545.956439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2545.992540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2546.047408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2546.664375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2546.904701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2546.939392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2546.982433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2547.024057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2547.071809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2547.118414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2547.152032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2547.223891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2547.271395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2547.300750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2547.342678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2547.372776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2547.408430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2547.454594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2547.491394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2552.514431] LNet: 125830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.528527] LNetError: 125830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2552.540508] LNet: Removed LNI 192.168.203.9@tcp [ 2552.555496] LNet: Removed LNI 192.168.203.9@tcp1 [ 2553.107273] Key type .llcrypt unregistered [ 2553.109173] Key type ._llcrypt unregistered [ 2570.296298] Key type ._llcrypt registered [ 2570.300875] Key type .llcrypt registered [ 2571.538871] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 01:47:00 (1788932820) [ 2571.870202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2571.882132] alg: No test for adler32 (adler32-zlib) [ 2572.867258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2572.942354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2572.985154] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2573.813340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2573.837376] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2574.691644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2574.799972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2575.840717] LNet: Removed LNI 192.168.203.9@tcp [ 2582.532051] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:47:11 (1788932831) [ 2584.152971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2586.104621] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:47:15 (1788932835) [ 2593.263490] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:47:22 (1788932842) [ 2600.509472] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:47:29 (1788932849) [ 2607.840445] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:47:36 (1788932856) [ 2615.371791] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:47:44 (1788932864) [ 2622.859653] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:47:51 (1788932871) [ 2630.794906] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:47:59 (1788932879) [ 2638.339739] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:48:07 (1788932887) [ 2646.649307] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:48:15 (1788932895) [ 2653.847513] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:48:22 (1788932902) [ 2660.886533] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:48:30 (1788932910) [ 2668.604217] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:48:37 (1788932917) [ 2675.359536] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:48:44 (1788932924) [ 2681.420706] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:48:50 (1788932930) [ 2687.597859] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:48:56 (1788932936) [ 2693.754371] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:49:02 (1788932942) [ 2700.769107] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:49:09 (1788932949) [ 2708.295210] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:49:17 (1788932957) [ 2708.381547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2708.422150] LNet: 137515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2708.426672] LNetError: 137515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2708.447455] LNet: Removed LNI 192.168.203.9@tcp2 [ 2708.480256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2708.643630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2708.684582] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2710.272590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2710.304590] LNet: 137663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2710.321760] LNetError: 137663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2710.352405] LNet: Removed LNI 192.168.203.9@tcp [ 2710.402255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2710.470727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2710.522446] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2718.502082] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:49:27 (1788932967) [ 2725.115656] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:49:34 (1788932974) [ 2734.162561] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:49:42 (1788932982) [ 2742.752743] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:49:51 (1788932991) [ 2750.104243] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:49:59 (1788932999) [ 2756.960677] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:50:06 (1788933006) [ 2764.159861] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:50:13 (1788933013) [ 2771.793701] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:50:20 (1788933020) [ 2779.978376] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:50:29 (1788933029) [ 2788.871802] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:50:37 (1788933037) [ 2796.032195] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:50:45 (1788933045) [ 2803.585931] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:50:52 (1788933052) [ 2810.555116] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2534 sec ========== 01:50:59 (1788933059) [ 2810.876990] LNet: 145129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2810.884244] LNetError: 145129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2810.902900] LNet: Removed LNI 192.168.203.9@tcp [ 2811.569947] Key type .llcrypt unregistered [ 2811.578656] Key type ._llcrypt unregistered [ 2823.765986] Key type ._llcrypt registered [ 2823.772421] Key type .llcrypt registered [ 2824.215129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2824.250314] alg: No test for adler32 (adler32-zlib) [ 2825.352644] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 2825.575858] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2827.280158] Key type lgssc registered [ 2828.520716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2898.056112] Lustre: Mounted lustre-client - version 2.17.58_39_gf1a8369 [ 2904.255467] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2913.451555] Lustre: Unmounted lustre-client [ 2935.036978] Key type lgssc unregistered [ 2935.279530] LNet: 148076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2935.289169] LNetError: 148076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2935.317795] LNet: Removed LNI 192.168.203.9@tcp [ 2936.180285] Key type .llcrypt unregistered [ 2936.191089] Key type ._llcrypt unregistered