[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 436655573 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003422] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008565] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011126] LSM: Security Framework initializing [ 0.013020] Yama: becoming mindful. [ 0.014040] SELinux: Initializing. [ 0.015088] *** VALIDATE selinux *** [ 0.022649] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027197] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030097] *** VALIDATE tmpfs *** [ 0.031460] *** VALIDATE proc *** [ 0.032260] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035235] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037023] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040015] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.044160] debug: unmapping init [mem 0xffffffffb2659000-0xffffffffb2660fff] [ 0.046744] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047691] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052058] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054010] ... event mask: 000000070000000f [ 0.055306] rcu: Hierarchical SRCU implementation. [ 0.057352] smp: Bringing up secondary CPUs ... [ 0.058622] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.062663] smp: Brought up 1 node, 4 CPUs [ 0.065019] smpboot: Max logical packages: 1 [ 0.066016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136590] node 0 deferred pages initialised in 67ms [ 0.140011] devtmpfs: initialized [ 0.141229] x86/mm: Memory block size: 128MB [ 0.143801] gcov: version magic: 0x41383552 [ 0.144582] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146234] pinctrl core: initialized pinctrl subsystem [ 0.147194] [ 0.147824] ************************************************************* [ 0.148015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149015] ** ** [ 0.150013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151015] ** ** [ 0.152011] ** This means that this kernel is built to expose internal ** [ 0.153016] ** IOMMU data structures, which may compromise security on ** [ 0.154014] ** your system. ** [ 0.155015] ** ** [ 0.156015] ** If you see this message and you are not debugging the ** [ 0.157013] ** kernel, report this immediately to your vendor! ** [ 0.158013] ** ** [ 0.159013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160017] ************************************************************* [ 0.161676] NET: Registered protocol family 16 [ 0.162480] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.166012] cpuidle: using governor menu [ 0.167962] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.171490] PCI: Using configuration type 1 for base access [ 0.173121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183103] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186126] cryptd: max_cpu_qlen set to 1000 [ 0.188206] ACPI: Added _OSI(Module Device) [ 0.189012] ACPI: Added _OSI(Processor Device) [ 0.190013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194990] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198096] ACPI: Interpreter enabled [ 0.199051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200011] ACPI: Using IOAPIC for interrupt routing [ 0.201101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202400] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.211702] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.212051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.213025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.214065] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.216453] acpiphp: Slot [2] registered [ 0.217153] acpiphp: Slot [5] registered [ 0.218122] acpiphp: Slot [6] registered [ 0.219112] acpiphp: Slot [3] registered [ 0.220107] acpiphp: Slot [4] registered [ 0.221120] acpiphp: Slot [7] registered [ 0.222126] acpiphp: Slot [8] registered [ 0.223126] acpiphp: Slot [9] registered [ 0.224111] acpiphp: Slot [10] registered [ 0.225110] acpiphp: Slot [11] registered [ 0.226105] acpiphp: Slot [12] registered [ 0.227108] acpiphp: Slot [13] registered [ 0.228097] acpiphp: Slot [14] registered [ 0.229111] acpiphp: Slot [15] registered [ 0.230171] acpiphp: Slot [16] registered [ 0.231079] acpiphp: Slot [17] registered [ 0.232088] acpiphp: Slot [18] registered [ 0.233088] acpiphp: Slot [19] registered [ 0.234070] acpiphp: Slot [20] registered [ 0.235094] acpiphp: Slot [21] registered [ 0.236127] acpiphp: Slot [22] registered [ 0.237089] acpiphp: Slot [23] registered [ 0.238104] acpiphp: Slot [24] registered [ 0.239096] acpiphp: Slot [25] registered [ 0.240158] acpiphp: Slot [26] registered [ 0.241116] acpiphp: Slot [27] registered [ 0.242066] acpiphp: Slot [28] registered [ 0.242935] acpiphp: Slot [29] registered [ 0.243061] acpiphp: Slot [30] registered [ 0.244078] acpiphp: Slot [31] registered [ 0.245073] PCI host bridge to bus 0000:00 [ 0.246028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.248025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.250021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.253023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.256024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.260045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.262228] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.266101] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.269342] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.277013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.282058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.285014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.288021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.290015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.294119] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.295759] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.299039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.302685] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.305933] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.316013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.321014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.325000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.330818] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.336013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.352832] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.365062] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.371011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.380015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.397013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.406060] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.408399] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.411413] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.413356] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.416230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.421057] iommu: Default domain type: Passthrough [ 0.423478] SCSI subsystem initialized [ 0.425088] ACPI: bus type USB registered [ 0.426134] usbcore: registered new interface driver usbfs [ 0.428085] usbcore: registered new interface driver hub [ 0.430094] usbcore: registered new device driver usb [ 0.431167] pps_core: LinuxPPS API ver. 1 registered [ 0.433016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.435049] PTP clock support registered [ 0.437067] EDAC MC: Ver: 3.0.0 [ 0.438379] PCI: Using ACPI for IRQ routing [ 0.439724] NetLabel: Initializing [ 0.441010] NetLabel: domain hash size = 128 [ 0.442008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.443073] NetLabel: unlabeled traffic allowed by default [ 0.445103] vgaarb: loaded [ 0.446196] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.447009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.452391] clocksource: Switched to clocksource kvm-clock [ 0.554555] VFS: Disk quotas dquot_6.6.0 [ 0.556607] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.560456] *** VALIDATE ramfs *** [ 0.562402] *** VALIDATE hugetlbfs *** [ 0.564804] pnp: PnP ACPI init [ 0.567810] pnp: PnP ACPI: found 6 devices [ 0.588503] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.592343] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.594829] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.597316] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.600219] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.603041] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.606201] NET: Registered protocol family 2 [ 0.609438] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.614983] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.619044] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.624612] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.627955] TCP: Hash tables configured (established 65536 bind 65536) [ 0.630774] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.633634] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.636283] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.639166] NET: Registered protocol family 1 [ 0.641570] RPC: Registered named UNIX socket transport module. [ 0.644283] RPC: Registered udp transport module. [ 0.646081] RPC: Registered tcp transport module. [ 0.648202] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.651027] NET: Registered protocol family 44 [ 0.652934] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.656159] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.659502] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.662888] PCI: CLS 0 bytes, default 64 [ 0.665207] Unpacking initramfs... [ 2.011854] debug: unmapping init [mem 0xffff99c43cc64000-0xffff99c43ffcffff] [ 2.015894] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.017577] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.019834] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.518970] Initialise system trusted keyrings [ 2.520364] Key type blacklist registered [ 2.522605] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.532479] zbud: loaded [ 2.535918] *** VALIDATE nfs *** [ 2.537363] *** VALIDATE nfs4 *** [ 2.539768] pstore: using deflate compression [ 2.543565] Platform Keyring initialized [ 2.630634] NET: Registered protocol family 38 [ 2.632037] Key type asymmetric registered [ 2.633175] Asymmetric key parser 'x509' registered [ 2.634562] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.636924] io scheduler mq-deadline registered [ 2.638404] io scheduler kyber registered [ 2.640016] io scheduler bfq registered [ 2.641756] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.643984] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.646133] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.648476] ACPI: Power Button [PWRF] [ 2.653144] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.659357] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.668769] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.697810] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.725437] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.729579] Non-volatile memory driver v1.3 [ 2.731175] Linux agpgart interface v0.103 [ 2.761245] virtio_blk virtio1: [vda] 149968 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.763885] vda: detected capacity change from 0 to 76783616 [ 2.775825] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.778233] vdb: detected capacity change from 0 to 1073741824 [ 2.784283] libphy: Fixed MDIO Bus: probed [ 2.788415] usbcore: registered new interface driver usbserial_generic [ 2.790467] usbserial: USB Serial support registered for generic [ 2.792421] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.795925] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.797393] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.799475] mousedev: PS/2 mouse device common for all mice [ 2.801762] rtc_cmos 00:05: RTC can wake from S4 [ 2.804208] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.807693] rtc_cmos 00:05: registered as rtc0 [ 2.809416] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.810196] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.814312] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.817192] intel_pstate: CPU model not supported [ 2.824043] hid: raw HID events driver (C) Jiri Kosina [ 2.825504] usbcore: registered new interface driver usbhid [ 2.827165] usbhid: USB HID core driver [ 2.828783] drop_monitor: Initializing network drop monitor service [ 2.831312] Initializing XFRM netlink socket [ 2.833294] NET: Registered protocol family 10 [ 2.835293] Segment Routing with IPv6 [ 2.836702] NET: Registered protocol family 17 [ 2.838785] mpls_gso: MPLS GSO support [ 2.844240] RAS: Correctable Errors collector initialized. [ 2.846414] AVX version of gcm_enc/dec engaged. [ 2.848256] AES CTR mode by8 optimization enabled [ 2.920592] sched_clock: Marking stable (2920571703, 0)->(3888790565, -968218862) [ 2.924850] registered taskstats version 1 [ 2.927411] Loading compiled-in X.509 certificates [ 2.929671] zswap: loaded using pool lzo/zbud [ 2.952618] Key type big_key registered [ 2.964156] Key type encrypted registered [ 2.965979] ima: No TPM chip found, activating TPM-bypass! [ 2.968121] ima: Allocated hash algorithm: sha1 [ 2.969339] ima: No architecture policies found [ 2.970516] evm: Initialising EVM extended attributes: [ 2.972018] evm: security.selinux [ 2.972900] evm: security.ima [ 2.973948] evm: security.capability [ 2.975363] evm: HMAC attrs: 0x1 [ 2.977941] rtc_cmos 00:05: setting system clock to 2026-09-30 09:27:53 UTC (1790760473) [ 2.984415] debug: unmapping init [mem 0xffffffffb3603000-0xffffffffb37fffff] [ 2.987541] debug: unmapping init [mem 0xffffffffb2382000-0xffffffffb2658fff] [ 2.997063] Write protecting the kernel read-only data: 28672k [ 3.000542] debug: unmapping init [mem 0xffffffffb0a03000-0xffffffffb0bfffff] [ 3.002919] debug: unmapping init [mem 0xffffffffb1314000-0xffffffffb13fffff] [ 3.029860] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.036742] systemd[1]: Detected virtualization kvm. [ 3.038420] systemd[1]: Detected architecture x86-64. [ 3.040767] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.065426] systemd[1]: No hostname configured. [ 3.067147] systemd[1]: Set hostname to . [ 3.068753] random: systemd: uninitialized urandom read (16 bytes read) [ 3.071328] systemd[1]: Initializing machine ID from random generator. [ 3.106575] random: ln: uninitialized urandom read (6 bytes read) [ 3.187222] random: systemd: uninitialized urandom read (16 bytes read) [ 3.190161] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.195107] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.199931] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.749467] device-mapper: uevent: version 1.0.3 [ 3.751740] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ 3.982761] random: fast init done [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.470163] virtio_net virtio0 ens2: renamed from eth0 [ 4.516042] scsi host0: ata_piix [ 4.557917] scsi host1: ata_piix [ 4.577498] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.580401] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.278568] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.490109] random: crng init done [ 9.492086] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.900659] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.142719] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.419484] SELinux: Disabled at runtime. [ 11.478714] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.488108] systemd[1]: Detected virtualization kvm. [ 11.490131] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.036301] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.039627] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.045078] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.049518] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.053053] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.065298] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.070969] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.227221] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.559863] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.897546] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.939709] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.046188] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.068744] EDAC sbridge: Ver: 1.1.2 [ 14.027162] Key type dns_resolver registered [ 14.327135] NFS: Registering the id_resolver key type [ 14.329526] Key type id_resolver registered [ 14.331339] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg653-client login: [ 41.616118] libcfs: loading out-of-tree module taints kernel. [ 41.742050] Key type ._llcrypt registered [ 41.743600] Key type .llcrypt registered [ 42.048991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 42.057438] alg: No test for adler32 (adler32-zlib) [ 43.124670] Lustre: Lustre: Build Version: 2.17.58_40_ge80375b [ 43.512424] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 45.160204] Key type lgssc registered [ 45.963712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.973015] hrtimer: interrupt took 7134860 ns [ 173.341543] Lustre: Mounted lustre-client - version 2.17.58_40_ge80375b [ 178.914452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 195.644493] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing check_logdir /tmp/testlogs/ [ 199.136683] Lustre: lustre-OST0000-osc-ffff99c4855b3000: disconnect after 23s idle [ 201.715462] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing yml_node [ 206.132881] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 208.487250] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 210.782351] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 212.518308] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 30 05:31:21 EDT 2026 [ 228.296689] Lustre: Unmounted lustre-client [ 266.295639] Key type lgssc unregistered [ 266.610770] LNet: 8430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 266.626381] LNetError: 8430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 266.656871] LNet: Removed LNI 192.168.206.53@tcp [ 267.599129] Key type .llcrypt unregistered [ 267.602797] Key type ._llcrypt unregistered [ 295.658858] Key type ._llcrypt registered [ 295.662704] Key type .llcrypt registered [ 295.922241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 295.944442] alg: No test for adler32 (adler32-zlib) [ 296.851525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 296.932614] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 296.967961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 300.823801] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 300.990417] LNet: 9304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.005171] LNetError: 9304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.021347] LNet: Removed LNI 192.168.206.53@tcp [ 301.599967] Key type .llcrypt unregistered [ 301.601726] Key type ._llcrypt unregistered [ 307.785176] Key type ._llcrypt registered [ 307.792393] Key type .llcrypt registered [ 309.160214] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 05:32:58 (1790760778) [ 309.540265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 309.557604] alg: No test for adler32 (adler32-zlib) [ 310.421737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.505585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.536375] LNet: 9717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.556981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.617547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 310.640551] LNet: 9724:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 317.918774] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 05:33:06 (1790760786) [ 318.054947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.116825] LNet: 10299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.157236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.233510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 325.032186] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 05:33:14 (1790760794) [ 325.365590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.382509] LNet: 10886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.431778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.544284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 333.344833] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 05:33:22 (1790760802) [ 333.445546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.492557] LNet: 11463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.542595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.665662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 339.813529] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 05:33:29 (1790760809) [ 340.006799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.030984] LNet: 12057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.055566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.179119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 340.522632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 345.828348] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 05:33:35 (1790760815) [ 346.047653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.098710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.166689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 346.708281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 347.087257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 352.870203] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 05:33:42 (1790760822) [ 353.001206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.031287] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.037462] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 353.062644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.151938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 358.773042] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 05:33:48 (1790760828) [ 359.091521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.159108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.248984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 359.295284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 359.668123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 359.709933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 360.044069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 360.089937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.449026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 360.487842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 360.848884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 360.880181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 361.168309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 361.198822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.560511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 361.587535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.919839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 361.953966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 362.376728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 362.413837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 362.783970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 362.832910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 363.183951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 363.207543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 363.504887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 363.546269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 363.931720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 363.976425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 370.171136] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 05:33:59 (1790760839) [ 370.343870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.372315] LNet: 14505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.379941] LNet: 14505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 370.406668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.497290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 370.554888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 376.191208] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 05:34:05 (1790760845) [ 376.353539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.397481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.462364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 376.492439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 383.012434] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 05:34:12 (1790760852) [ 383.286228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.324544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.401563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 383.448873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 389.826052] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 05:34:19 (1790760859) [ 390.052676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.116412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.246990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 390.311870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 396.717442] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 05:34:25 (1790760865) [ 396.985425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.078980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.259430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 397.291747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 404.374193] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 05:34:33 (1790760873) [ 404.478084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.518331] LNet: 17435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.524105] LNet: 17435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 404.580038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.797806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 404.849459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 412.051566] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 05:34:41 (1790760881) [ 412.283942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.323302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.423388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 412.865764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.634088] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 05:34:48 (1790760888) [ 419.802450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.858185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.053353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 420.284185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.293864] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 05:34:56 (1790760896) [ 427.512490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.543602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.673869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 427.871586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 433.896590] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 05:35:03 (1790760903) [ 433.995136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.031728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.584570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.827654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.765835] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 05:35:10 (1790760910) [ 441.864515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.928649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.058272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.123862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 448.717452] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 05:35:17 (1790760917) [ 448.825933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.875732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.996192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 449.051249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.902792] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 05:35:24 (1790760924) [ 455.019667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.086653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.165234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 461.074974] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 05:35:30 (1790760930) [ 461.256212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.316577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.388847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 467.524451] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 05:35:36 (1790760936) [ 467.865803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.913217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.000723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 468.553061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 474.939347] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 05:35:44 (1790760944) [ 475.166562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.181434] LNet: 23469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.187574] LNet: 23469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 475.202387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.250850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 475.285627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 481.388532] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 05:35:50 (1790760950) [ 481.690630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.756133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.869772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 481.927571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 488.019643] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 05:35:57 (1790760957) [ 488.231312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.267690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.330894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 488.372503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 494.537583] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 05:36:03 (1790760963) [ 494.779333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.850296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.940193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 494.970963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 500.867867] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 05:36:10 (1790760970) [ 501.109688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.171577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.213595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 501.279629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 501.328305] LNetError: 25826:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 501.384972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 501.435837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 510.779552] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 05:36:18 (1790760978) [ 510.910585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.061632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.153590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PTpY1P7NaI [ 511.257349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PTpY1P7NaI [ 511.307638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PTpY1P7NaI [ 521.236523] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 05:36:29 (1790760989) [ 521.529200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.656669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.790103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 521.896443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 521.992626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 529.449533] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 05:36:38 (1790760998) [ 529.572662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.644541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.818104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 529.874611] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 531.787767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 533.329228] Lustre: Lustre: Build Version: 2.17.58_40_ge80375b [ 535.224355] Key type lgssc registered [ 536.355767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 539.378242] LNetError: 27589:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.206.53@tcp99 has no available nets [ 542.001060] Key type lgssc unregistered [ 542.427561] LNetError: 28696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.453109] LNet: Removed LNI 192.168.206.53@tcp [ 542.956228] Key type .llcrypt unregistered [ 542.961407] Key type ._llcrypt unregistered [ 549.352953] Key type ._llcrypt registered [ 549.354552] Key type .llcrypt registered [ 550.500285] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 05:36:59 (1790761019) [ 551.088253] Key type .llcrypt unregistered [ 551.096073] Key type ._llcrypt unregistered [ 556.624685] Key type ._llcrypt registered [ 556.629235] Key type .llcrypt registered [ 556.811499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 556.831282] alg: No test for adler32 (adler32-zlib) [ 557.751544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.821795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 557.849892] LNet: 29616:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 557.860898] LNet: 29616:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 557.870518] LNet: 29616:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 557.886331] LNet: Message forwarding will be enabled in 2s [ 560.992123] LNet: Message forwarding enabled [ 561.226982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 561.247099] LNet: Message forwarding will stop in 2s to 6s [ 565.090424] LNet: Message forwarding disabled [ 565.654047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 565.667900] LNet: 29714:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 565.673252] LNet: 29714:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 565.680504] LNet: 29714:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 565.695759] LNet: Message forwarding will be enabled in 2s [ 565.852314] LNet: 29727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.392468] Key type .llcrypt unregistered [ 566.395679] Key type ._llcrypt unregistered [ 576.727808] Key type ._llcrypt registered [ 576.730024] Key type .llcrypt registered [ 578.471119] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 05:37:26 (1790761046) [ 578.916700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 578.936632] alg: No test for adler32 (adler32-zlib) [ 579.835954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.921949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 585.747727] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 05:37:35 (1790761055) [ 585.935338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.951150] LNet: 31126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.968183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.005931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 586.042036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 586.067298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 586.092688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 586.116534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 586.140534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 586.166394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 586.191905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 586.225138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 586.266017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 586.295343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 586.332875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 586.374281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 586.412422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 586.446496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 586.484507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 586.512263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 586.545338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 586.579716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 586.610804] LNetError: 31169:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 592.676885] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 05:37:41 (1790761061) [ 592.843653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.865417] LNet: 31740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.893891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.962991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 599.680472] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 05:37:48 (1790761068) [ 601.698434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg653-server [ 603.411513] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 05:37:52 (1790761072) [ 603.520555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.543875] LNet: 32665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 603.562715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.627437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 603.662929] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 604.463597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[54-58]@tcp [ 604.484993] LNet: 32804:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 604.546623] LNetError: 32673:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.54@tcp is being used as a gateway but routing feature is not turned on [ 604.574949] LNetError: 32673:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.54@tcp has gone from up to down [ 610.870499] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 05:37:59 (1790761079) [ 612.441364] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg653-server [ 614.367673] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 05:38:03 (1790761083) [ 614.491293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.531517] LNet: 33733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.546692] LNetError: 33733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.584849] LNet: Removed LNI 192.168.206.53@tcp [ 614.631865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.705659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 614.723270] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 615.570285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[54-59/2]@tcp [ 615.598933] LNet: 33856:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 615.604639] LNet: 33856:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 615.621079] LNetError: 33743:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.54@tcp is being used as a gateway but routing feature is not turned on [ 615.630584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[54-59/2]@tcp [ 615.634881] LNetError: 33743:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 615.634916] LNetError: 33743:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.54@tcp has gone from up to down [ 615.634921] LNetError: 33743:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 621.860186] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 05:38:11 (1790761091) [ 621.946112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.965408] LNet: 34443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.971801] LNetError: 34443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.986880] LNet: Removed LNI 192.168.206.53@tcp [ 622.013396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.097960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 622.142942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.182892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.255794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 622.304448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.354764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.427506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 622.525375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.560287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.618396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 622.712407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.780286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.848173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 622.927746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.954681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.007789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.065939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.097329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.139944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 623.209854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.244229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.299917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.383490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.415203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.465356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 623.535881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.567283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.637712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.728546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.758562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.807885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 623.844579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.868333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.913471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 629.418962] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 05:38:18 (1790761098) [ 631.059545] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg653-server [ 632.863206] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 05:38:22 (1790761102) [ 634.815917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg653-server [ 636.283809] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 05:38:25 (1790761105) [ 636.739487] LNet: 35849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.747581] LNet: 35849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 637.474451] Key type .llcrypt unregistered [ 637.478942] Key type ._llcrypt unregistered [ 643.729828] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 643.935198] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 643.939197] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 643.945570] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 646.601381] Key type ._llcrypt registered [ 646.613142] Key type .llcrypt registered [ 646.920195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 646.955133] alg: No test for adler32 (adler32-zlib) [ 647.897317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.957446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.992627] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 648.659779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 648.677614] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 650.866714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 650.894526] LNet: Removed LNI 10.1.2.3@tcp [ 651.040515] LNet: 36565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.050488] LNetError: 36565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.076818] LNet: Removed LNI 192.168.206.53@tcp [ 651.958433] Key type .llcrypt unregistered [ 651.962095] Key type ._llcrypt unregistered [ 663.870883] Key type ._llcrypt registered [ 663.872341] Key type .llcrypt registered [ 665.220651] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 05:38:54 (1790761134) [ 665.764318] Key type .llcrypt unregistered [ 665.766749] Key type ._llcrypt unregistered [ 672.230121] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 672.447516] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 672.452548] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 672.459217] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 675.041804] Key type ._llcrypt registered [ 675.047892] Key type .llcrypt registered [ 675.252232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 675.277376] alg: No test for adler32 (adler32-zlib) [ 676.204125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.287639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.305136] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 677.090571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 677.119462] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 679.677865] LNet: 38057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.690981] LNetError: 38057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 679.696844] LNet: Removed LNI 10.1.2.3@tcp [ 679.711548] LNet: Removed LNI 192.168.206.53@tcp [ 680.411532] Key type .llcrypt unregistered [ 680.416781] Key type ._llcrypt unregistered [ 692.742323] Key type ._llcrypt registered [ 692.744534] Key type .llcrypt registered [ 693.999874] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 05:39:23 (1790761163) [ 694.796505] Key type .llcrypt unregistered [ 694.804953] Key type ._llcrypt unregistered [ 702.517857] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 702.840255] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 702.846245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 702.849815] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 705.572435] Key type ._llcrypt registered [ 705.576808] Key type .llcrypt registered [ 705.832879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 705.866332] alg: No test for adler32 (adler32-zlib) [ 706.803900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.880490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 706.919685] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 708.718565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 708.749524] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 710.686876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 710.704604] LNet: Removed LNI 10.1.2.3@tcp [ 710.718993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 710.752125] LNet: Removed LNI 10.1.2.31@tcp [ 710.776337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 710.817425] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 711.403958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 711.420040] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 712.111410] LNet: 39971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.117339] LNetError: 39971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 712.121751] LNet: Removed LNI 10.1.2.31@tcp [ 712.145090] LNet: Removed LNI 10.1.2.3@tcp [ 712.689410] Key type .llcrypt unregistered [ 712.691136] Key type ._llcrypt unregistered [ 723.992392] Key type ._llcrypt registered [ 723.994626] Key type .llcrypt registered [ 724.990748] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 05:39:54 (1790761194) [ 725.615738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 725.634355] alg: No test for adler32 (adler32-zlib) [ 726.589734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.646297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 726.681932] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 727.647515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.691590] LNet: 40922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.701821] LNetError: 40922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.741229] LNet: Removed LNI 192.168.206.53@tcp [ 727.782950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.838222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.53@tcp [ 727.859682] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 728.806697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.53@tcp [ 728.859910] LNet: Removed LNI 192.168.206.53@tcp [ 728.989701] LNet: 41057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.767508] Key type .llcrypt unregistered [ 729.770674] Key type ._llcrypt unregistered [ 742.624760] Key type ._llcrypt registered [ 742.629113] Key type .llcrypt registered [ 743.900576] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 05:40:13 (1790761213) [ 745.846206] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg653-server [ 747.813250] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 05:40:16 (1790761216) [ 748.497913] Key type .llcrypt unregistered [ 748.503843] Key type ._llcrypt unregistered [ 754.524566] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.781932] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.788076] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.794435] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.446769] Key type ._llcrypt registered [ 757.450530] Key type .llcrypt registered [ 757.662948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.681490] alg: No test for adler32 (adler32-zlib) [ 758.601286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.677251] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 759.666215] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 762.082696] LNet: 42889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.103363] LNetError: 42889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.111945] LNet: Removed LNI 192.168.206.53@tcp1 [ 762.128429] LNet: Removed LNI 10.1.2.3@tcp2 [ 762.824455] Key type .llcrypt unregistered [ 762.830904] Key type ._llcrypt unregistered [ 777.055823] Key type ._llcrypt registered [ 777.058289] Key type .llcrypt registered [ 778.045468] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 05:40:47 (1790761247) [ 778.674315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 778.689684] alg: No test for adler32 (adler32-zlib) [ 779.621154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.692807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 779.725917] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 780.575611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.605608] LNet: 43856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.611991] LNetError: 43856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.624160] LNet: Removed LNI 192.168.206.53@tcp [ 780.654320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.755102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 780.787450] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 787.800834] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 05:40:56 (1790761256) [ 787.920937] LNet: 44441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.928398] LNetError: 44441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.947348] LNet: Removed LNI 192.168.206.53@tcp [ 788.470645] Key type .llcrypt unregistered [ 788.475499] Key type ._llcrypt unregistered [ 794.541960] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 794.831069] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 794.843777] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 794.860597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 797.481229] Key type ._llcrypt registered [ 797.483609] Key type .llcrypt registered [ 797.785848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 797.820251] alg: No test for adler32 (adler32-zlib) [ 798.854890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.987895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.026370] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 800.012868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 800.051313] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.070441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.088788] LNet: 45188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.094299] LNetError: 45188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.098974] LNet: Removed LNI 10.1.2.3@tcp [ 802.113182] LNet: Removed LNI 192.168.206.53@tcp [ 802.155416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.252538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 802.289095] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 802.301043] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.853359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.884422] LNet: 45215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.894740] LNetError: 45215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.901652] LNetError: 45215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 802.909270] LNet: Removed LNI 10.1.2.3@tcp [ 802.918071] LNet: Removed LNI 192.168.206.53@tcp [ 802.936449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.017691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 803.056352] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 803.064548] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 803.575281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 803.593955] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 804.483319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.499397] LNet: 45377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.506738] LNetError: 45377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 804.515319] LNetError: 45377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 804.522222] LNet: Removed LNI 10.1.2.3@tcp [ 804.524239] LNet: Removed LNI 192.168.206.53@tcp [ 804.549776] LNet: Removed LNI 192.168.206.53@tcp2 [ 804.570063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.622418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 804.668300] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 804.677542] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 804.684025] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 814.070557] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 05:41:22 (1790761282) [ 814.196496] LNet: 45968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.202879] LNetError: 45968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 814.212267] LNetError: 45968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 814.221138] LNet: Removed LNI 10.1.2.3@tcp [ 814.222614] LNet: Removed LNI 192.168.206.53@tcp [ 814.264786] LNet: Removed LNI 192.168.206.53@tcp2 [ 815.299252] Key type .llcrypt unregistered [ 815.302456] Key type ._llcrypt unregistered [ 822.420297] Key type ._llcrypt registered [ 822.426283] Key type .llcrypt registered [ 822.872374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 822.884444] alg: No test for adler32 (adler32-zlib) [ 823.737371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.802406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.841438] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 824.919589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.959199] LNet: 46366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.971222] LNetError: 46366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.006788] LNet: Removed LNI 192.168.206.53@tcp [ 825.039441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.168696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 825.213470] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 825.861547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.910315] LNet: 46396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.924148] LNetError: 46396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.002163] LNet: Removed LNI 192.168.206.53@tcp [ 826.067941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.265261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 826.329788] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 827.121877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.165788] LNet: 46427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.174358] LNetError: 46427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.213921] LNet: Removed LNI 192.168.206.53@tcp [ 827.276482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.384566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 827.457296] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 827.998909] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 828.126210] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 828.130955] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 828.136199] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 830.815930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.851030] LNet: 46506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.859576] LNetError: 46506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.881840] LNet: Removed LNI 192.168.206.53@tcp [ 830.923521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.058578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 831.095161] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 838.780379] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 05:41:47 (1790761307) [ 838.976303] LNet: 47100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.982894] LNetError: 47100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.029300] LNet: Removed LNI 192.168.206.53@tcp [ 839.637118] Key type .llcrypt unregistered [ 839.643389] Key type ._llcrypt unregistered [ 846.223296] Key type ._llcrypt registered [ 846.225309] Key type .llcrypt registered [ 846.381955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 846.394696] alg: No test for adler32 (adler32-zlib) [ 847.280683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.359725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 847.380126] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 848.241025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.280430] LNet: 47499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.293878] LNetError: 47499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.316813] LNet: Removed LNI 192.168.206.53@tcp [ 848.339574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.473945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 848.504191] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 854.769123] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 05:42:04 (1790761324) [ 854.952760] LNet: 48086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.971757] LNetError: 48086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.007138] LNet: Removed LNI 192.168.206.53@tcp [ 855.838293] Key type .llcrypt unregistered [ 855.842617] Key type ._llcrypt unregistered [ 862.862179] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 863.280430] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 863.289459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 863.302465] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 865.859342] Key type ._llcrypt registered [ 865.866837] Key type .llcrypt registered [ 866.118692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.134611] alg: No test for adler32 (adler32-zlib) [ 867.115609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.168887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.194046] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 868.376863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.433730] LNet: 48538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.439298] LNetError: 48538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.460770] LNet: Removed LNI 192.168.206.53@tcp [ 868.491545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.563914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 868.589279] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 876.352198] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 05:42:25 (1790761345) [ 876.444867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.474473] LNet: 49129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.486566] LNetError: 49129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.504194] LNet: Removed LNI 192.168.206.53@tcp [ 876.559867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.712871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 883.531198] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 05:42:32 (1790761352) [ 883.689232] LNet: 49712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.459766] Key type .llcrypt unregistered [ 884.467555] Key type ._llcrypt unregistered [ 890.288602] Key type ._llcrypt registered [ 890.290979] Key type .llcrypt registered [ 890.486491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.502510] alg: No test for adler32 (adler32-zlib) [ 891.539888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.614107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 891.647932] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 892.743570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.761610] LNet: 50112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.769706] LNetError: 50112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.788902] LNet: Removed LNI 192.168.206.53@tcp [ 892.823662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.911731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 892.961513] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 899.765443] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 05:42:48 (1790761368) [ 900.023309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.073023] LNet: 50706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.086577] LNetError: 50706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.111139] LNet: Removed LNI 192.168.206.53@tcp [ 900.155724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.267935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 906.311812] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 05:42:55 (1790761375) [ 908.091635] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 909.964226] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 05:42:58 (1790761378) [ 911.863099] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 914.271539] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 05:43:02 (1790761382) [ 916.221106] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 918.437873] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 05:43:07 (1790761387) [ 920.086155] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 921.730915] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 05:43:10 (1790761390) [ 923.637823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 925.586737] Lustre: DEBUG MARKER: == sanity-lnet test 163: Check import of net with mixed IPv6/IPv4 explicit NIDs ========================================================== 05:43:14 (1790761394) [ 925.739943] LNet: 52723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.770791] Key type .llcrypt unregistered [ 926.785285] Key type ._llcrypt unregistered [ 932.956146] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 933.114946] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 933.121868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 933.131575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 935.749250] Key type ._llcrypt registered [ 935.755713] Key type .llcrypt registered [ 936.081689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 936.109706] alg: No test for adler32 (adler32-zlib) [ 937.090433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.179823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-163-expected.yaml [ 937.214030] LNet: Added LNI 2001:db8:0:f101::1@tcp [8/256/0/180] [ 937.226900] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 937.361862] LNet: 53066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.374752] LNetError: 53066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 937.391249] LNet: Removed LNI 2001:db8:0:f101::1@tcp [ 937.418399] LNet: Removed LNI 10.1.2.3@tcp1 [ 938.248550] Key type .llcrypt unregistered [ 938.252563] Key type ._llcrypt unregistered [ 950.795451] Key type ._llcrypt registered [ 950.800324] Key type .llcrypt registered [ 952.089522] Lustre: DEBUG MARKER: == sanity-lnet test 164: Check no issue when reading 'nid' in all scenarios ========================================================== 05:43:41 (1790761421) [ 952.705244] Key type .llcrypt unregistered [ 952.709988] Key type ._llcrypt unregistered [ 958.529425] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.756555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.408400] Key type ._llcrypt registered [ 961.411618] Key type .llcrypt registered [ 961.652513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.673142] alg: No test for adler32 (adler32-zlib) [ 962.648536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.378036] Lustre: DEBUG MARKER: Check malformed nid fails [ 964.494375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 964.596335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.670553] LNet: 54303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.700207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.369819] Lustre: DEBUG MARKER: Check wrong net in nid fails [ 966.483948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 966.540257] LNetError: 54447:0:(api-ni.c:4164:lnet_dyn_add_ni()) NID network tcp2 does not match configured network tcp1 [ 966.598419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.620970] LNet: 54452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.646526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.730948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 966.762655] LNetError: 54460:0:(config.c:1721:lnet_inet_select()) ksocklnd: failed to find UP interface with IP address 192.168.206.53@tcp1 [ 966.797757] LNetError: Error -100 starting up LNI tcp [ 966.846960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.902297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.243958] Lustre: DEBUG MARKER: Check 'nid' cannot be used in ip2nets [ 968.443145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 968.516403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.534082] LNet: 54618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.538935] LNet: 54618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 968.563131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.915176] Lustre: DEBUG MARKER: Check MR works [ 970.004126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 970.056812] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 970.073879] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 970.222749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.251862] LNetError: 54780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 970.261721] LNet: Removed LNI 10.1.2.3@tcp [ 970.275925] LNet: Removed LNI 192.168.206.53@tcp [ 970.309274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.588534] Lustre: DEBUG MARKER: Check nid/interfaces order doesn't matter [ 971.660294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 971.698082] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 971.724490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp [ 971.805780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.826464] LNet: 54943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.831542] LNet: 54943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 971.839787] LNetError: 54943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.845418] LNetError: 54943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 971.858064] LNet: Removed LNI 192.168.206.53@tcp [ 971.872815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.507496] Key type .llcrypt unregistered [ 972.516267] Key type ._llcrypt unregistered [ 982.936689] Key type ._llcrypt registered [ 982.941106] Key type .llcrypt registered [ 984.228818] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing net ========================================================== 05:44:13 (1790761453) [ 984.950712] Key type .llcrypt unregistered [ 984.960362] Key type ._llcrypt unregistered [ 991.962189] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 992.309293] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 994.987386] Key type ._llcrypt registered [ 994.990395] Key type .llcrypt registered [ 995.168510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 995.185824] alg: No test for adler32 (adler32-zlib) [ 996.094023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.590617] Lustre: DEBUG MARKER: Different CPTs [ 997.814113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 997.852815] LNet: Added LNI 192.168.206.53@tcp [8/512/0/180] [ 997.901081] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 998.066237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.083954] LNet: 56197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.089780] LNetError: 56197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.094982] LNet: Removed LNI 10.1.2.3@tcp [ 998.098152] LNetError: 56197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.113084] LNet: Removed LNI 192.168.206.53@tcp [ 998.143688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.932170] Lustre: DEBUG MARKER: lnd_tunables not read if no tunables [ 1000.037100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1000.113128] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1000.248334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.280522] LNet: 56356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.292521] LNetError: 56356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.315805] LNet: Removed LNI 192.168.206.53@tcp [ 1000.355019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.870994] Lustre: DEBUG MARKER: lnd_tunables cannot be different [ 1002.017903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1002.073977] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1002.082813] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1002.272183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.299661] LNet: 56520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.312393] LNetError: 56520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.321227] LNet: Removed LNI 10.1.2.3@tcp [ 1002.326960] LNetError: 56520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.356048] LNet: Removed LNI 192.168.206.53@tcp [ 1002.377547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.433363] Lustre: DEBUG MARKER: Wrong tunables (lnd_tunables) read from NI if first is empty [ 1004.633614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1004.697205] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1004.706847] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1004.722184] LNet: 56665:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1004.974715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.001528] LNet: 56691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.010385] LNetError: 56691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.025162] LNet: Removed LNI 10.1.2.3@tcp [ 1005.026951] LNetError: 56691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.059126] LNet: Removed LNI 192.168.206.53@tcp [ 1005.096858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.032295] Lustre: DEBUG MARKER: Tunables can be set directly in the net [ 1007.146285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1007.214930] LNet: Added LNI 192.168.206.53@tcp [8/256/0/182] [ 1007.234207] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1007.448911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.468546] LNet: 56855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.486396] LNetError: 56855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.493421] LNet: Removed LNI 10.1.2.3@tcp [ 1007.497177] LNetError: 56855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.519803] LNet: Removed LNI 192.168.206.53@tcp [ 1007.555279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.052208] Lustre: DEBUG MARKER: Per-NI lnd tunables combined with net-level tunables [ 1009.143174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1009.194952] LNet: Added LNI 192.168.206.53@tcp [8/256/0/182] [ 1009.387438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.478714] LNet: 57017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.503823] LNetError: 57017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.555568] LNet: Removed LNI 192.168.206.53@tcp [ 1009.642198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.456319] Lustre: DEBUG MARKER: Per-NI tunables combined with net-level lnd tunables [ 1011.581890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1011.621416] LNet: Added LNI 192.168.206.53@tcp [8/256/0/182] [ 1011.843873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.862357] LNet: 57180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.868230] LNetError: 57180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.883722] LNet: Removed LNI 192.168.206.53@tcp [ 1011.927384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.797658] Lustre: DEBUG MARKER: Import net block without 'local NI(s):' uses resolver [ 1013.905726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1013.944298] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1015.697767] Lustre: DEBUG MARKER: import --show of net block without 'local NI(s):' [ 1015.937330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --show [ 1016.023546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1016.060974] LNet: 57481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.069821] LNetError: 57481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.093128] LNet: Removed LNI 192.168.206.53@tcp [ 1016.119282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.553655] Lustre: DEBUG MARKER: Show warning if we try to set different tunables (not lnd_tunables) for NIs of a same net [ 1017.793160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1017.858750] LNet: Added LNI 192.168.206.53@tcp [8/256/0/182] [ 1017.876796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1017.882226] LNet: 57624:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1018.051494] LNet: Removed LNI 10.1.2.3@tcp [ 1018.060727] LNet: Removed LNI 192.168.206.53@tcp [ 1018.839053] Key type .llcrypt unregistered [ 1018.844682] Key type ._llcrypt unregistered [ 1031.130121] Key type ._llcrypt registered [ 1031.132901] Key type .llcrypt registered [ 1032.572660] Lustre: DEBUG MARKER: == sanity-lnet test 171: Check CPTs and tunables parsing when importing net (old-api) ========================================================== 05:45:01 (1790761501) [ 1033.321680] Key type .llcrypt unregistered [ 1033.326448] Key type ._llcrypt unregistered [ 1040.163840] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1040.485715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1043.249187] Key type ._llcrypt registered [ 1043.253231] Key type .llcrypt registered [ 1043.553940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1043.576500] alg: No test for adler32 (adler32-zlib) [ 1044.485942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.844830] Lustre: DEBUG MARKER: Different CPTs [ 1045.918557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1045.961253] LNet: Added LNI 192.168.206.53@tcp [8/512/0/180] [ 1045.967511] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1045.969325] LNet: 58865:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1046.077959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.102789] LNet: 58891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.114499] LNetError: 58891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.125919] LNet: Removed LNI 10.1.2.3@tcp [ 1046.132826] LNetError: 58891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.162257] LNet: Removed LNI 192.168.206.53@tcp [ 1046.185798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.560035] Lustre: DEBUG MARKER: lnd_tunables not read if no tunables [ 1047.756584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.791777] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1047.894694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.934579] LNet: 59051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.939993] LNetError: 59051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.955818] LNet: Removed LNI 192.168.206.53@tcp [ 1047.987455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1049.792151] Lustre: DEBUG MARKER: lnd_tunables cannot be different [ 1049.868054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1049.924557] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1049.930128] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1049.934924] LNet: 59189:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1050.048537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1050.093225] LNet: 59214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.105357] LNetError: 59214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1050.114160] LNet: Removed LNI 10.1.2.3@tcp [ 1050.117237] LNetError: 59214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.136370] LNet: Removed LNI 192.168.206.53@tcp [ 1050.169130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.752506] Lustre: DEBUG MARKER: Wrong tunables (lnd_tunables) read from NI if first is empty [ 1051.852312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1051.884689] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1051.898321] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1051.901335] LNet: 59351:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1052.197917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1052.215244] LNet: 59385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.222091] LNetError: 59385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1052.227665] LNet: Removed LNI 10.1.2.3@tcp [ 1052.229830] LNetError: 59385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.245054] LNet: Removed LNI 192.168.206.53@tcp [ 1052.260260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.631794] Lustre: DEBUG MARKER: Tunables can be set directly in the net [ 1054.091570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1054.129823] LNet: Added LNI 192.168.206.53@tcp [8/256/0/182] [ 1054.141732] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1054.259184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.294753] LNet: 59550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.300723] LNetError: 59550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1054.307744] LNet: Removed LNI 10.1.2.3@tcp [ 1054.309273] LNetError: 59550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.325417] LNet: Removed LNI 192.168.206.53@tcp [ 1054.345608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.805420] Lustre: DEBUG MARKER: Per-NI lnd tunables combined with net-level tunables [ 1055.920324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1056.004943] LNet: Added LNI 192.168.206.53@tcp [8/256/0/182] [ 1056.166755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1056.188659] LNet: 59712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.194542] LNetError: 59712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.209898] LNet: Removed LNI 192.168.206.53@tcp [ 1056.264815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.332100] Lustre: DEBUG MARKER: Per-NI tunables combined with net-level lnd tunables [ 1058.420611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1058.453709] LNet: Added LNI 192.168.206.53@tcp [8/256/0/182] [ 1058.658229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.692589] LNet: 59874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.699522] LNetError: 59874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.726135] LNet: Removed LNI 192.168.206.53@tcp [ 1058.754846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.539907] Lustre: DEBUG MARKER: Show warning if we try to set different tunables (not lnd_tunables) for NIs of a same net [ 1060.723096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1060.774489] LNet: Added LNI 192.168.206.53@tcp [8/256/0/182] [ 1060.785055] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1060.792602] LNet: 60015:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1061.034766] LNet: Removed LNI 10.1.2.3@tcp [ 1061.059739] LNet: Removed LNI 192.168.206.53@tcp [ 1061.886847] Key type .llcrypt unregistered [ 1061.900091] Key type ._llcrypt unregistered [ 1075.007493] Key type ._llcrypt registered [ 1075.018925] Key type .llcrypt registered [ 1076.449373] Lustre: DEBUG MARKER: == sanity-lnet test 172: Check CPTs and tunables parsing when importing ip2nets ========================================================== 05:45:45 (1790761545) [ 1076.979425] Key type .llcrypt unregistered [ 1076.981741] Key type ._llcrypt unregistered [ 1083.250283] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1083.505237] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1086.201160] Key type ._llcrypt registered [ 1086.204884] Key type .llcrypt registered [ 1086.473711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.496398] alg: No test for adler32 (adler32-zlib) [ 1087.408398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1087.510427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1087.537607] LNet: Added LNI 192.168.206.53@tcp [8/432/0/182] [ 1087.545871] LNet: Added LNI 10.1.2.3@tcp1 [8/402/0/181] [ 1087.751899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1087.800267] LNet: 61162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.812460] LNetError: 61162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.826651] LNet: Removed LNI 192.168.206.53@tcp [ 1087.843565] LNet: Removed LNI 10.1.2.3@tcp1 [ 1087.876388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1087.967938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1088.022202] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1088.036921] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/181] [ 1088.187505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1088.207397] LNet: Removed LNI 192.168.206.53@tcp [ 1088.216604] LNet: Removed LNI 10.1.2.3@tcp1 [ 1088.245482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.301895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1088.324449] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1088.328352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1088.406617] LNet: 61217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.411567] LNet: 61217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1088.416950] LNetError: 61217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1088.421725] LNetError: 61217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1088.427857] LNet: Removed LNI 10.1.2.3@tcp [ 1088.434683] LNet: Removed LNI 192.168.206.53@tcp [ 1088.944686] Key type .llcrypt unregistered [ 1088.946467] Key type ._llcrypt unregistered [ 1101.349204] Key type ._llcrypt registered [ 1101.356146] Key type .llcrypt registered [ 1103.073663] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 05:46:11 (1790761571) [ 1104.301439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1106.083797] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 05:46:15 (1790761575) [ 1106.444507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1106.460473] alg: No test for adler32 (adler32-zlib) [ 1107.449189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1107.546168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1107.597754] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1108.381652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.53 [ 1108.403061] LNet: 62463:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1108.411829] LNet: Removed LNI 192.168.206.53@tcp [ 1115.164280] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 05:46:24 (1790761584) [ 1115.263359] LNet: 63024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.277247] LNetError: 63024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.302363] LNet: Removed LNI 192.168.206.53@tcp [ 1115.994416] Key type .llcrypt unregistered [ 1115.999412] Key type ._llcrypt unregistered [ 1122.708175] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1123.057295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1125.735826] Key type ._llcrypt registered [ 1125.739490] Key type .llcrypt registered [ 1125.901710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1125.926691] alg: No test for adler32 (adler32-zlib) [ 1127.395285] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1134.222111] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 05:46:43 (1790761603) [ 1134.337304] LNet: 63934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.357204] LNetError: 63934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1134.393356] LNet: Removed LNI 10.1.2.3@tcp [ 1135.253577] Key type .llcrypt unregistered [ 1135.263696] Key type ._llcrypt unregistered [ 1141.334869] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1141.600053] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1141.606988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1141.612397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1143.139077] Key type ._llcrypt registered [ 1143.142348] Key type .llcrypt registered [ 1143.328331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1143.342629] alg: No test for adler32 (adler32-zlib) [ 1144.309741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1150.244256] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 05:46:59 (1790761619) [ 1152.084337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1153.757093] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 05:47:02 (1790761622) [ 1153.904947] LNet: 65110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.915205] LNetError: 65110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1153.950839] LNet: Removed LNI 10.1.2.3@tcp [ 1154.589571] Key type .llcrypt unregistered [ 1154.592616] Key type ._llcrypt unregistered [ 1160.310513] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1160.566481] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1160.571556] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1160.578622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1163.302287] Key type ._llcrypt registered [ 1163.305274] Key type .llcrypt registered [ 1163.501459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1163.517339] alg: No test for adler32 (adler32-zlib) [ 1164.403880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.474918] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1173.265486] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 05:47:22 (1790761642) [ 1173.957422] LNet: 66344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.965401] LNetError: 66344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1174.001443] LNet: Removed LNI 10.1.2.3@tcp [ 1174.673469] Key type .llcrypt unregistered [ 1174.675390] Key type ._llcrypt unregistered [ 1181.454649] Key type ._llcrypt registered [ 1181.458441] Key type .llcrypt registered [ 1181.662868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1181.694929] alg: No test for adler32 (adler32-zlib) [ 1182.694793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1182.728871] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1192.184685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1192.907545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_interrupt -r 1 [ 1192.961906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_interrupt -r 1 [ 1193.011891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1193.034614] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1193.169373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.197314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.226242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.260417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.286890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1193.316800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.350535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.383284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.414123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.433726] LNet: 66782:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.454876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.485572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.516560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.542936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.582715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.612728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.648330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1194.107718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_dropped -r 1 [ 1194.148946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_dropped -r 1 [ 1194.178453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1194.201061] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1194.515636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.551224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1194.585745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1194.629304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1194.661140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1194.696241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1194.737280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1194.763804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1194.791131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1194.812734] LNet: 66893:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1194.829665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.857937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.889836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.918287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.943713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.970463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.021200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1195.490404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_aborted -r 1 [ 1195.526715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_aborted -r 1 [ 1195.564684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1195.588346] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1195.840919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1195.869103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1195.898883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.934587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.965706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1196.006607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.044318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.073676] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1196.077923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.136512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.179790] LNet: 67004:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.217614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.295271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.359336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.397289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.422973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.448303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.481134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1197.013191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_no_route -r 1 [ 1197.041472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_no_route -r 1 [ 1197.078452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1197.307784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.332579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1197.360923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.385686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.413610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1197.450361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.482896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.511996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.541686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.569700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.601995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.630857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.669367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.698863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.729820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.760755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1198.184736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_timeout -r 1 [ 1198.217596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_timeout -r 1 [ 1198.245299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1198.264508] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1198.268860] LNet: Skipped 2 previous similar messages [ 1198.486873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1198.514775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1198.544789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.572337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.599259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1198.627718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.661435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.691659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.720622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.735818] LNet: 67227:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1198.742136] LNet: 67227:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1198.754643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.778201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1198.806045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.830853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.857532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.890335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.917233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1199.403853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_error -r 1 [ 1199.432685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_error -r 1 [ 1199.464367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1199.644510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.663407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.686620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.716533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.740656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1199.775663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.798559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.843517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.885637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.919857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.947284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.976657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.010612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.034919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.064116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.093720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1200.402890] LNet: 67379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.407213] LNetError: 67379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.423807] LNet: Removed LNI 192.168.206.53@tcp [ 1200.892546] Key type .llcrypt unregistered [ 1200.895345] Key type ._llcrypt unregistered [ 1219.223405] Key type ._llcrypt registered [ 1219.225487] Key type .llcrypt registered [ 1220.835925] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 05:48:09 (1790761689) [ 1222.160697] Key type .llcrypt unregistered [ 1222.164209] Key type ._llcrypt unregistered [ 1228.461900] Key type ._llcrypt registered [ 1228.465704] Key type .llcrypt registered [ 1228.674753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1228.693849] alg: No test for adler32 (adler32-zlib) [ 1229.730686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1229.775665] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1237.917218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1249.509196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.562771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1249.616945] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1251.240693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_interrupt -r 1 [ 1251.293428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_interrupt -r 1 [ 1251.357716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_interrupt -r 1 [ 1251.404489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_interrupt -r 1 [ 1251.454549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_interrupt -r 1 [ 1251.529174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_interrupt -r 1 [ 1251.585934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_interrupt -r 1 [ 1251.637297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_interrupt -r 1 [ 1251.677342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1251.697705] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1251.704724] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1252.019783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.078569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1252.109215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1252.139772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1252.174632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1252.211072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.241847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.268900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.298991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.321191] LNet: 68846:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1252.341693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.362567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.397992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.441736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.485451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1252.526505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.568037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1253.193069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_dropped -r 1 [ 1253.274228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_dropped -r 1 [ 1253.333529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_dropped -r 1 [ 1253.376833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_dropped -r 1 [ 1253.433367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_dropped -r 1 [ 1253.486580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_dropped -r 1 [ 1253.532818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_dropped -r 1 [ 1253.590366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_dropped -r 1 [ 1253.637261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1253.669441] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1253.676261] LNet: Skipped 2 previous similar messages [ 1253.979801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.014911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1254.047201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.108580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.148104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1254.204584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.255503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.322063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.356790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.373697] LNet: 68970:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1254.388432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1254.413595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.439339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.465438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1254.500256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.534372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1254.568776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1255.058745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_aborted -r 1 [ 1255.119560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_aborted -r 1 [ 1255.154849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_aborted -r 1 [ 1255.186930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_aborted -r 1 [ 1255.240925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_aborted -r 1 [ 1255.291857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_aborted -r 1 [ 1255.327719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_aborted -r 1 [ 1255.369457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_aborted -r 1 [ 1255.409091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1255.446600] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1255.453275] LNet: Skipped 2 previous similar messages [ 1255.654217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1255.682841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1255.711138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1255.735965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1255.771504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1255.804914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1255.844768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1255.876727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1255.914804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1255.936866] LNet: 69093:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1255.963411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1256.009957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1256.043977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.086409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.122105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.155372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.186753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1256.727792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_no_route -r 1 [ 1256.771165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_no_route -r 1 [ 1256.809957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_no_route -r 1 [ 1256.888611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_no_route -r 1 [ 1256.956851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_no_route -r 1 [ 1257.013573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_no_route -r 1 [ 1257.073807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_no_route -r 1 [ 1257.121309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_no_route -r 1 [ 1257.177432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1257.447337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1257.477402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1257.502363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1257.528858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1257.562612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1257.603437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1257.653385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1257.691517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1257.727319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1257.767692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1257.794423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1257.822544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1257.851749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1257.893760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1257.927302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1257.960833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1258.414910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_timeout -r 1 [ 1258.443589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_timeout -r 1 [ 1258.477802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_timeout -r 1 [ 1258.506939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_timeout -r 1 [ 1258.559144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_timeout -r 1 [ 1258.612963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_timeout -r 1 [ 1258.671431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_timeout -r 1 [ 1258.717323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_timeout -r 1 [ 1258.753540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1258.777430] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1258.788797] LNet: Skipped 5 previous similar messages [ 1259.006108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1259.028138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1259.049590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1259.080568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1259.103828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1259.127834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.150318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.181493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.210641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.225910] LNet: 69340:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.231907] LNet: 69340:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1259.249895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.289707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.323452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.363125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1259.408882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.439836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1259.491942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.053640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_error -r 1 [ 1260.085464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_error -r 1 [ 1260.140747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_error -r 1 [ 1260.192625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_error -r 1 [ 1260.256245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_error -r 1 [ 1260.321820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_error -r 1 [ 1260.388193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_error -r 1 [ 1260.449759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_error -r 1 [ 1260.515441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1260.821340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.856745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1260.894311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.934757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.973726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1261.009560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.035826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.069965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1261.117342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1261.151608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1261.178462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.205594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.234344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.271438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.316407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1261.355585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1266.265303] LNet: 69539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1266.270333] LNetError: 69539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.281401] LNet: Removed LNI 192.168.206.53@tcp [ 1266.290059] LNet: Removed LNI 192.168.206.53@tcp1 [ 1266.581448] Key type .llcrypt unregistered [ 1266.583272] Key type ._llcrypt unregistered [ 1281.539493] Key type ._llcrypt registered [ 1281.545722] Key type .llcrypt registered [ 1282.849514] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 05:49:11 (1790761751) [ 1284.010286] Key type .llcrypt unregistered [ 1284.012900] Key type ._llcrypt unregistered [ 1291.829543] Key type ._llcrypt registered [ 1291.832454] Key type .llcrypt registered [ 1292.148597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1292.165443] alg: No test for adler32 (adler32-zlib) [ 1293.230325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1293.272726] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1303.445950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1304.483667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_dropped -r 1 [ 1304.516080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_dropped -r 1 [ 1304.565634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1304.610201] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1304.838624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.875314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1304.902676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.935601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.974548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1305.007794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.034407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.071735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.129985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.152571] LNet: 70787:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.172658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.229787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.280147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.319456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.368568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.398155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.428303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1305.823593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_error -r 1 [ 1305.846965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_error -r 1 [ 1305.870409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1305.890987] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1306.086984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.123907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1306.161611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1306.198835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1306.264718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1306.303324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1306.338560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1306.419823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1306.473388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1306.508934] LNet: 70898:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1306.541643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.601656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.654401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.703778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.760698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.819283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.881889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1307.567983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_timeout -r 1 [ 1307.632742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_timeout -r 1 [ 1307.673171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1307.696629] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1307.945588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1307.978551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1308.020537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1308.071149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1308.108420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1308.136474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1308.165786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1308.193691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1308.264605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1308.292973] LNet: 71009:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1308.333341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1308.388302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1308.471708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.510057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1308.569099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1308.628544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1308.676654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1309.122958] LNet: 71050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.128941] LNetError: 71050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.145850] LNet: Removed LNI 192.168.206.53@tcp [ 1309.617727] Key type .llcrypt unregistered [ 1309.620078] Key type ._llcrypt unregistered [ 1323.907237] Key type ._llcrypt registered [ 1323.908866] Key type .llcrypt registered [ 1325.316468] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 05:49:54 (1790761794) [ 1326.071593] Key type .llcrypt unregistered [ 1326.075961] Key type ._llcrypt unregistered [ 1332.000280] Key type ._llcrypt registered [ 1332.002668] Key type .llcrypt registered [ 1332.292250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1332.321437] alg: No test for adler32 (adler32-zlib) [ 1333.330729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1333.362462] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1340.771899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1348.470907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1348.538336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1348.582403] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1349.723643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_dropped -r 1 [ 1349.769340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e remote_dropped -r 1 [ 1349.817522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_dropped -r 1 [ 1349.841786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e remote_dropped -r 1 [ 1349.883302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e remote_dropped -r 1 [ 1349.910402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e remote_dropped -r 1 [ 1349.957483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e remote_dropped -r 1 [ 1350.014420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e remote_dropped -r 1 [ 1350.048582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1350.067336] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1350.075423] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.153@tcp [ 1350.264861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.297738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1350.325982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.363422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.394134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1350.444158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1350.479576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1350.514247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1350.564272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1350.586495] LNet: 72515:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1350.629958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1350.672290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1350.717563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.749401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1350.786933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1350.820584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1350.846278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1350.872557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.356490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_error -r 1 [ 1351.386985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e remote_error -r 1 [ 1351.418546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_error -r 1 [ 1351.457377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e remote_error -r 1 [ 1351.488830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e remote_error -r 1 [ 1351.516546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e remote_error -r 1 [ 1351.546790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e remote_error -r 1 [ 1351.580942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e remote_error -r 1 [ 1351.618953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1351.637817] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1351.643746] LNet: Skipped 2 previous similar messages [ 1351.819827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1351.839652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1351.863980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1351.885558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1351.917799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1351.951362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1351.986445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1352.027773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1352.061524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1352.088441] LNet: 72641:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1352.110371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1352.152927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1352.189283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1352.226418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1352.273867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1352.329824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1352.369245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1352.406691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1352.996854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_timeout -r 1 [ 1353.041480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e remote_timeout -r 1 [ 1353.098589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_timeout -r 1 [ 1353.143161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e remote_timeout -r 1 [ 1353.186865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e remote_timeout -r 1 [ 1353.247759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e remote_timeout -r 1 [ 1353.293297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e remote_timeout -r 1 [ 1353.351157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e remote_timeout -r 1 [ 1353.407807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1353.449245] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1353.462101] LNet: Skipped 1 previous similar message [ 1353.731823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1353.767880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1353.796145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1353.822285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1353.857337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1353.885336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1353.909338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1353.931921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1353.954465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1353.969597] LNet: 72766:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1353.991120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1354.022891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1354.057417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.098188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1354.139076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1354.196066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1354.242528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1354.294408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.712880] LNet: 72844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.721783] LNetError: 72844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.730304] LNet: Removed LNI 192.168.206.53@tcp [ 1358.751130] LNet: Removed LNI 192.168.206.53@tcp1 [ 1359.131742] Key type .llcrypt unregistered [ 1359.135720] Key type ._llcrypt unregistered [ 1375.510116] Key type ._llcrypt registered [ 1375.511961] Key type .llcrypt registered [ 1376.977085] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 05:50:45 (1790761845) [ 1378.858704] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1380.549352] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 05:50:49 (1790761849) [ 1381.429602] Key type .llcrypt unregistered [ 1381.435112] Key type ._llcrypt unregistered [ 1386.704551] Key type ._llcrypt registered [ 1386.706750] Key type .llcrypt registered [ 1386.867049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1386.890865] alg: No test for adler32 (adler32-zlib) [ 1387.781973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1387.812298] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1394.696700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1395.295838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e network_timeout -r 1 [ 1395.327864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e network_timeout -r 1 [ 1395.359961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1395.377359] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1395.565787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1395.604374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1395.642895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1395.670828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1395.704878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1395.737959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1395.773487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1395.797773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1395.844961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1395.878505] LNet: 74385:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1395.910649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1395.948171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1395.987760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1396.000250] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1396.032750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1396.068335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1396.099424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1396.330792] LNet: 74424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.335847] LNetError: 74424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.349755] LNet: Removed LNI 192.168.206.53@tcp [ 1396.698764] Key type .llcrypt unregistered [ 1396.703314] Key type ._llcrypt unregistered [ 1413.896159] Key type ._llcrypt registered [ 1413.900654] Key type .llcrypt registered [ 1414.178394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1414.189919] alg: No test for adler32 (adler32-zlib) [ 1415.202959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1415.239689] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1422.933433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1432.044131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.093687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1432.117620] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1433.854865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e network_timeout -r 1 [ 1433.923718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e network_timeout -r 1 [ 1433.986691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e network_timeout -r 1 [ 1434.038403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e network_timeout -r 1 [ 1434.120791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e network_timeout -r 1 [ 1434.197590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e network_timeout -r 1 [ 1434.264744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e network_timeout -r 1 [ 1434.333387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e network_timeout -r 1 [ 1434.368974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1434.385584] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1434.718494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1434.730476] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1434.738207] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.153@tcp [ 1434.768288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1434.804793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1434.837616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1434.876503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1434.923225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1434.966933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1435.025301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1435.066642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1435.097561] LNet: 75299:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1435.126309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1435.185876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1435.235270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1435.289303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1435.338652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1435.385491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1439.999820] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1440.380990] LNet: 75373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.388295] LNetError: 75373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.399873] LNet: Removed LNI 192.168.206.53@tcp [ 1440.418902] LNet: Removed LNI 192.168.206.53@tcp1 [ 1440.805514] Key type .llcrypt unregistered [ 1440.807992] Key type ._llcrypt unregistered [ 1456.913335] Key type ._llcrypt registered [ 1456.923717] Key type .llcrypt registered [ 1458.455689] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 05:52:07 (1790761927) [ 1458.942368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1458.967346] alg: No test for adler32 (adler32-zlib) [ 1459.961853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.045938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1460.079937] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1460.831766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1460.849186] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1461.841242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1461.925746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1462.014556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1462.054505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1462.089850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1462.128399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1462.175141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.53@tcp [ 1462.205490] LNetError: 76469:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.206.53@tcp to recovery [ 1463.072358] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1463.072626] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1465.120593] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1465.762611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1469.220316] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1469.590186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1469.642025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1469.749876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.792037] LNet: 76577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.803326] LNetError: 76577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.880219] LNet: Removed LNI 192.168.206.53@tcp [ 1470.882470] LNetError: 76577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.892803] LNet: Removed LNI 192.168.206.53@tcp1 [ 1470.942551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.019424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.057045] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1472.045299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1472.069566] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1473.447665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1473.501375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1473.668494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1473.748774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1473.799116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1473.838822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1473.867229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1473.921349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.53@tcp [ 1473.944572] LNetError: 76849:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.206.53@tcp to recovery [ 1474.080382] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1476.426281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1480.226535] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1480.244714] LNet: Skipped 1 previous similar message [ 1484.527118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1484.565457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1484.616753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1492.129578] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 05:52:41 (1790761961) [ 1492.246427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.295226] LNet: 77551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.307707] LNetError: 77551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.335593] LNet: Removed LNI 192.168.206.53@tcp [ 1492.357425] LNet: Removed LNI 192.168.206.53@tcp1 [ 1492.432743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.485127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1492.520348] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1493.176187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1493.202898] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1494.093567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1494.163611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1494.190408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1494.235213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1494.277726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1494.312841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1494.352843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.53@tcp --health 0 [ 1494.446214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1494.504997] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.53@tcp [ 1495.521881] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1495.543983] LNet: Skipped 1 previous similar message [ 1509.916502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1510.066527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1510.095927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1510.137839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.53@tcp --health 500 [ 1510.231885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1517.912301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1517.961604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.990107] LNet: 78056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.996187] LNetError: 78056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.001309] LNetError: 78056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1518.007605] LNet: Removed LNI 192.168.206.53@tcp [ 1518.027976] LNet: Removed LNI 192.168.206.53@tcp1 [ 1518.059256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.123704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1518.178424] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1518.852782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.872103] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1519.741699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1519.789389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.895683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1519.946741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1520.004113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1520.039970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1520.088247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1520.118506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.53@tcp --health 0 [ 1521.184374] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1521.188542] LNet: Skipped 3 previous similar messages [ 1521.499869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1531.809563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1531.864568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1531.914929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1540.056799] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 05:53:28 (1790762008) [ 1540.596846] LNet: 79105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.608818] LNetError: 79105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.619332] LNetError: 79105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1540.626586] LNet: Removed LNI 192.168.206.53@tcp [ 1540.646736] LNet: Removed LNI 192.168.206.53@tcp1 [ 1541.419732] Key type .llcrypt unregistered [ 1541.428262] Key type ._llcrypt unregistered [ 1549.441083] Key type ._llcrypt registered [ 1549.442815] Key type .llcrypt registered [ 1549.627768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1549.640498] alg: No test for adler32 (adler32-zlib) [ 1550.643864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1550.696546] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1560.252573] Lustre: DEBUG MARKER: Initial discovery [ 1560.355372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.153@tcp [ 1562.539492] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1562.832269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1562.880093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.153@tcp [ 1562.910081] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1562.937770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1563.001352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.153@tcp -r 1 -m GET -l 3 [ 1563.041287] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1563.059675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.153@tcp -r 1 -m GET -e local_error [ 1563.123755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.153@tcp -r 1 -m PUT -l 6 [ 1564.441443] Lustre: DEBUG MARKER: Force oleg653-client.virtnet to discover 192.168.206.153@tcp (in background) [ 1564.596165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.153@tcp [ 1565.897084] Lustre: DEBUG MARKER: Force oleg653-server to discover 192.168.206.53@tcp [ 1567.648715] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1567.969953] Lustre: DEBUG MARKER: Wait for 79865 [ 1569.453404] Lustre: DEBUG MARKER: Finished wait on 79865 [ 1569.745624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1569.793982] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1569.824727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1570.136544] LNet: 80315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.145393] LNetError: 80315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.164457] LNet: Removed LNI 192.168.206.53@tcp [ 1570.562669] Key type .llcrypt unregistered [ 1570.567587] Key type ._llcrypt unregistered [ 1590.463522] Key type ._llcrypt registered [ 1590.470721] Key type .llcrypt registered [ 1591.993069] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 05:54:20 (1790762060) [ 1592.601498] Key type .llcrypt unregistered [ 1592.604215] Key type ._llcrypt unregistered [ 1599.004213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1599.225269] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1599.244556] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1599.257251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1601.920971] Key type ._llcrypt registered [ 1601.923212] Key type .llcrypt registered [ 1602.147498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1602.164192] alg: No test for adler32 (adler32-zlib) [ 1603.122767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.210458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.248627] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1604.087722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1604.136948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1613.234762] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 05:54:42 (1790762082) [ 1613.460923] LNet: 82382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.470738] LNetError: 82382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1613.482773] LNet: Removed LNI 10.1.2.3@tcp [ 1613.513142] LNet: Removed LNI 192.168.206.53@tcp [ 1614.282632] Key type .llcrypt unregistered [ 1614.289204] Key type ._llcrypt unregistered [ 1622.893125] Key type ._llcrypt registered [ 1622.895623] Key type .llcrypt registered [ 1623.141560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1623.161325] alg: No test for adler32 (adler32-zlib) [ 1624.157464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.241359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.310282] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1624.863667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1624.894341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1626.145553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.53@tcp [ 1627.232724] LNetError: Unexpected error -22 connecting to 192.168.206.53@tcp at host 192.168.206.53:988 [ 1633.030554] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 05:55:02 (1790762102) [ 1633.141346] LNet: 83513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.147449] LNetError: 83513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1633.164621] LNet: Removed LNI 10.1.2.3@tcp [ 1633.199436] LNet: Removed LNI 192.168.206.53@tcp [ 1633.837981] Key type .llcrypt unregistered [ 1633.851400] Key type ._llcrypt unregistered [ 1639.582539] Key type ._llcrypt registered [ 1639.587499] Key type .llcrypt registered [ 1639.824280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1639.845432] alg: No test for adler32 (adler32-zlib) [ 1640.765122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.829945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1640.849417] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1641.558886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1641.588370] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 1642.387712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.53@tcp1 --nid 192.168.206.53@tcp2 [ 1646.868507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp1 192.168.206.53@tcp2 [ 1646.895440] LNetError: 84350:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.53@tcp1 to 192.168.206.53@tcp2 - Route table may be misconfigured [ 1646.924360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp2 192.168.206.53@tcp1 [ 1653.869943] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 05:55:23 (1790762123) [ 1654.130461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.169422] LNet: 84923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.176969] LNetError: 84923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.187320] LNet: Removed LNI 192.168.206.53@tcp1 [ 1655.208855] LNet: Removed LNI 192.168.206.53@tcp2 [ 1655.241682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.306718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.354801] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1656.138585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1656.161948] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1657.070930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1657.122760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.53@tcp [ 1657.168816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp1 192.168.206.53@tcp1 [ 1657.220815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.53@tcp -d 192.168.206.53@tcp -e network_timeout [ 1657.257257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -e network_timeout [ 1657.304266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -e network_timeout [ 1657.348694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -e network_timeout [ 1657.394879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1657.423347] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1657.451796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1657.586053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1657.702987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1658.338696] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1664.071425] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 05:55:33 (1790762133) [ 1664.179408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.211443] LNet: 85776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.221541] LNetError: 85776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.234483] LNetError: 85776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1664.249393] LNet: Removed LNI 192.168.206.53@tcp [ 1665.319231] LNet: Removed LNI 192.168.206.53@tcp1 [ 1665.384822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.530770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1665.908847] LNet: 85822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.495515] Key type .llcrypt unregistered [ 1666.498393] Key type ._llcrypt unregistered [ 1678.277528] Key type ._llcrypt registered [ 1678.280691] Key type .llcrypt registered [ 1679.681867] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 05:55:48 (1790762148) [ 1680.157925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.178872] alg: No test for adler32 (adler32-zlib) [ 1681.206469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.870955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1684.865337] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 05:55:53 (1790762153) [ 1684.965150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.028129] LNet: 86908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.062801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.164054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1685.199225] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1686.064924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1686.094211] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1686.965809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1686.996150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp1 [ 1687.030216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp1 [ 1694.497082] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 05:56:03 (1790762163) [ 1696.343448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg653-server [ 1698.488708] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 05:56:07 (1790762167) [ 1700.559380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg653-server [ 1702.772169] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 05:56:11 (1790762171) [ 1704.930645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg653-server [ 1706.708472] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 05:56:15 (1790762175) [ 1709.275398] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg653-server [ 1711.493076] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 05:56:20 (1790762180) [ 1713.438993] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg653-server [ 1715.548495] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 05:56:24 (1790762184) [ 1717.275770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg653-server [ 1718.988221] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 05:56:28 (1790762188) [ 1721.065525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg653-server [ 1722.626922] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 05:56:31 (1790762191) [ 1724.529463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg653-server [ 1727.589465] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 05:56:35 (1790762195) [ 1729.912558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg653-server [ 1731.727677] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 05:56:40 (1790762200) [ 1731.845852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.887178] LNet: 90788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.896754] LNetError: 90788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.906737] LNet: Removed LNI 192.168.206.53@tcp [ 1732.967523] LNet: Removed LNI 192.168.206.53@tcp1 [ 1733.000226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.110725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.141867] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1734.128932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1734.269238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1735.683351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.711375] LNet: 90951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.717630] LNetError: 90951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.733735] LNetError: 90951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1735.767815] LNet: Removed LNI 192.168.206.53@tcp [ 1735.807845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.903679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1735.980117] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1736.779521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1737.050448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.087893] LNetError: 91096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.096772] LNet: Removed LNI 192.168.206.53@tcp [ 1737.157535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.232960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1737.254877] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1737.922307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1738.010571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1738.248135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.284888] LNet: 91244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.297156] LNet: 91244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1739.389835] LNet: Removed LNI 192.168.206.53@tcp [ 1739.441666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.529876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1739.573531] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1740.582131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1740.790954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.831739] LNetError: 91389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.843928] LNetError: 91389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1740.856838] LNet: Removed LNI 192.168.206.53@tcp [ 1740.884777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.951322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1740.975953] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1741.517706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1741.646917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1741.916066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.951712] LNet: Removed LNI 192.168.206.53@tcp [ 1743.003258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.092768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1743.146442] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1744.090925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1744.377268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.438750] LNet: 91682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.449358] LNet: 91682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1745.511490] LNet: Removed LNI 192.168.206.53@tcp [ 1745.572924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.657400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1745.690262] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1746.469962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1746.567492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1746.888484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.916988] LNetError: 91829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.929986] LNetError: 91829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1746.942660] LNet: Removed LNI 192.168.206.53@tcp [ 1746.956239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.994756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1747.017660] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1747.786751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1748.060771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.160937] LNet: Removed LNI 192.168.206.53@tcp [ 1749.201355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.304336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.351469] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1750.141873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1750.242847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1750.523180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.590275] LNet: Removed LNI 192.168.206.53@tcp [ 1751.626128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.679171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1751.702870] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1752.345916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1752.595861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.612816] LNet: 92268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.624895] LNet: 92268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1753.706473] LNet: Removed LNI 192.168.206.53@tcp [ 1753.757913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.807524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1753.836662] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1754.500039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1754.556288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1754.863824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.943348] LNet: Removed LNI 192.168.206.53@tcp [ 1755.972646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.007730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1756.025080] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1756.761970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1757.122703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.161356] LNetError: 92561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.168107] LNetError: 92561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1757.194273] LNet: Removed LNI 192.168.206.53@tcp [ 1757.234731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.313625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.335300] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1758.038876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1758.072395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.099808] LNet: Removed LNI 192.168.206.53@tcp [ 1758.142426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.193819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.220190] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1758.830959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1758.885994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.909368] LNet: Removed LNI 192.168.206.53@tcp [ 1758.923175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.967899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1759.030840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.061452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.102868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1759.170630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.203769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.249343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1759.314430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.339542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.376871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1759.397781] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1760.076527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.112076] LNet: Removed LNI 192.168.206.53@tcp [ 1760.126321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.185177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1766.821182] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 05:57:15 (1790762235) [ 1767.108603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.168348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.238396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1767.255656] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/180] [ 1768.326069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.354044] LNet: Removed LNI 192.168.206.53@tcp231 [ 1768.387772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.479219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1768.525236] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/180] [ 1768.531659] LNet: 93728:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.003575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1769.041163] LNet: Removed LNI 192.168.206.53@tcp231 [ 1769.059186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1769.082652] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/180] [ 1770.530279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.555970] LNet: 93872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.559372] LNet: 93872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1770.573802] LNet: Removed LNI 192.168.206.53@tcp231 [ 1770.596642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.681983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1770.736042] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/0] [ 1770.751763] LNet: 93880:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1771.170680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1771.206962] LNet: Removed LNI 192.168.206.53@tcp231 [ 1771.245284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1771.308080] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/0] [ 1772.629584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.661960] LNet: Removed LNI 192.168.206.53@tcp231 [ 1772.682733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.759364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1772.789376] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/60] [ 1772.800974] LNet: 94032:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1773.326792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1773.384460] LNet: Removed LNI 192.168.206.53@tcp231 [ 1773.400536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1773.454691] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/60] [ 1774.607298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.640050] LNetError: 94175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.649760] LNetError: 94175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1774.662097] LNet: Removed LNI 192.168.206.53@tcp231 [ 1774.681352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.769666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1774.808369] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/180] [ 1774.815169] LNet: 94183:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1775.500130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1775.561914] LNet: Removed LNI 192.168.206.53@tcp231 [ 1775.582607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1775.603113] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/180] [ 1784.998057] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 05:57:33 (1790762253) [ 1785.140450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.230068] LNet: Removed LNI 192.168.206.53@tcp231 [ 1785.297627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.425623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1785.449194] LNet: Added LNI 192.168.206.53@tcp232 [8/256/0/180] [ 1786.431437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1786.470623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1786.499783] LNet: Removed LNI 192.168.206.53@tcp232 [ 1786.520310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1786.557455] LNet: Added LNI 192.168.206.53@tcp232 [8/256/0/180] [ 1787.423624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1787.482791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1787.512718] LNet: Removed LNI 192.168.206.53@tcp232 [ 1787.553132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1787.586624] LNet: Added LNI 192.168.206.53@tcp232 [8/256/0/180] [ 1788.312794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1788.358609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1788.393550] LNet: Removed LNI 192.168.206.53@tcp232 [ 1788.411232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1788.443500] LNet: Added LNI 192.168.206.53@tcp232 [8/256/0/180] [ 1789.108475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1789.135968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1789.161784] LNet: Removed LNI 192.168.206.53@tcp232 [ 1789.189878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1796.430661] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 05:57:45 (1790762265) [ 1797.693388] Key type .llcrypt unregistered [ 1797.697395] Key type ._llcrypt unregistered [ 1803.833380] Key type ._llcrypt registered [ 1803.840923] Key type .llcrypt registered [ 1804.237192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1804.285058] alg: No test for adler32 (adler32-zlib) [ 1805.243397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1805.295923] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1813.598141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1822.814451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.872094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1822.912294] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1824.759386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_interrupt -r 2 [ 1824.812905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_interrupt -r 2 [ 1824.915397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_interrupt -r 2 [ 1824.975383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_interrupt -r 2 [ 1825.029844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_interrupt -r 2 [ 1825.083624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_interrupt -r 2 [ 1825.142968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_interrupt -r 2 [ 1825.224374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_interrupt -r 2 [ 1825.278357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1825.309567] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1825.314633] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1825.599502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1825.634749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1825.694363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1825.769707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1825.829950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1825.876781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1825.930319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1825.990511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1826.087205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1826.123125] LNet: 96682:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1826.158735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1826.202855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1826.264743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1826.306077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1826.352308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1826.391663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1826.960044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_dropped -r 2 [ 1827.012903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_dropped -r 2 [ 1827.057296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_dropped -r 2 [ 1827.090587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_dropped -r 2 [ 1827.118295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_dropped -r 2 [ 1827.148646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_dropped -r 2 [ 1827.196675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_dropped -r 2 [ 1827.236520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_dropped -r 2 [ 1827.279635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1827.297416] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1827.522405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1827.551851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1827.582908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1827.609771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1827.636661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1827.669944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1827.707356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1827.748157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1827.801779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1827.819966] LNet: 96805:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1827.837426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1827.868412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1827.909358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1827.945377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1827.999657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1828.029671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1828.595962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_aborted -r 2 [ 1828.637894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_aborted -r 2 [ 1828.662841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_aborted -r 2 [ 1828.688194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_aborted -r 2 [ 1828.714692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_aborted -r 2 [ 1828.768892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_aborted -r 2 [ 1828.818683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_aborted -r 2 [ 1828.858505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_aborted -r 2 [ 1828.892755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1828.913540] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1829.147767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1829.178901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1829.206230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1829.231399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1829.259579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1829.290450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1829.316966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1829.346590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1829.375081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1829.390500] LNet: 96927:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1829.404313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1829.427910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1829.466622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1829.491694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1829.517976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1829.542271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1829.966906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_no_route -r 2 [ 1829.995379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_no_route -r 2 [ 1830.026882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_no_route -r 2 [ 1830.061616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_no_route -r 2 [ 1830.112375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_no_route -r 2 [ 1830.153165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_no_route -r 2 [ 1830.201253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_no_route -r 2 [ 1830.235521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_no_route -r 2 [ 1830.281959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1830.510495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1830.532670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1830.559506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1830.593487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1830.616645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1830.639312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1830.665086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1830.704458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1830.753312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1830.789372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1830.841891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1830.879844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1830.926140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1830.988343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1831.030443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1831.588621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_timeout -r 2 [ 1831.620555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_timeout -r 2 [ 1831.652042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_timeout -r 2 [ 1831.681685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_timeout -r 2 [ 1831.708514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_timeout -r 2 [ 1831.740447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_timeout -r 2 [ 1831.768730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_timeout -r 2 [ 1831.813538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_timeout -r 2 [ 1831.847866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1831.869635] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1831.875961] LNet: Skipped 1 previous similar message [ 1832.100657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1832.149546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1832.186678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1832.219687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1832.253378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1832.291459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1832.329971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1832.398713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1832.448437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1832.474501] LNet: 97172:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1832.487496] LNet: 97172:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1832.509371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1832.557651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1832.612422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1832.651521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1832.694122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1832.721787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1833.286754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_dropped -r 2 [ 1833.322538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e remote_dropped -r 2 [ 1833.363488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_dropped -r 2 [ 1833.402495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e remote_dropped -r 2 [ 1833.439965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e remote_dropped -r 2 [ 1833.490480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e remote_dropped -r 2 [ 1833.546696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e remote_dropped -r 2 [ 1833.585712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e remote_dropped -r 2 [ 1833.629917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1833.666743] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.153@tcp [ 1833.921659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1833.966569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1834.005318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1834.030310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1834.059477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1834.091577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1834.131861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1834.161582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1834.191752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1834.229991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1834.263520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1834.302066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1834.334732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1834.387768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1834.436930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1840.347246] LNet: 97368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.356872] LNetError: 97368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.368072] LNet: Removed LNI 192.168.206.53@tcp [ 1840.381638] LNet: Removed LNI 192.168.206.53@tcp1 [ 1840.759716] Key type .llcrypt unregistered [ 1840.763467] Key type ._llcrypt unregistered [ 1858.556395] Key type ._llcrypt registered [ 1858.559155] Key type .llcrypt registered [ 1859.799731] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 05:58:48 (1790762328) [ 1860.488310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1860.508109] alg: No test for adler32 (adler32-zlib) [ 1861.423790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.481877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1861.502375] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1862.296507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1862.346638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.53@tcp [ 1862.386462] LNetError: 98344:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.206.53@tcp to recovery [ 1862.483378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.53@tcp [ 1862.496532] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1862.589478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1868.808897] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 05:58:58 (1790762338) [ 1868.988219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.030865] LNet: 98932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.046189] LNetError: 98932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.073072] LNet: Removed LNI 192.168.206.53@tcp [ 1869.119772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.196964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1869.232397] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1869.913402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1869.929771] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1870.897840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.53@tcp [ 1870.913766] LNetError: 99200:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.206.53@tcp to recovery [ 1871.208232] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1872.154953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.53@tcp [ 1879.979425] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 05:59:09 (1790762349) [ 1881.672391] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg653-server [ 1883.681546] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 05:59:12 (1790762352) [ 1886.024812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg653-server [ 1888.093940] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 05:59:17 (1790762357) [ 1890.497275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg653-server [ 1892.363543] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 05:59:21 (1790762361) [ 1892.425160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1892.456198] LNet: 100814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.468048] LNetError: 100814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1892.475277] LNet: Removed LNI 192.168.206.53@tcp [ 1892.492463] LNet: Removed LNI 192.168.206.53@tcp1 [ 1892.517339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1892.589228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1892.607146] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1893.348746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1893.463211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1893.525552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1893.673989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1893.725480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1893.843794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1902.281710] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 05:59:30 (1790762370) [ 1902.327246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1902.384200] LNet: 101547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.396580] LNetError: 101547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.403549] LNetError: 101547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1902.458263] LNet: Removed LNI 192.168.206.53@tcp [ 1902.497051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.578280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1902.607212] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1909.918973] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 05:59:38 (1790762378) [ 1910.040977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1910.097914] LNet: 102252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.122224] LNetError: 102252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.153408] LNet: Removed LNI 192.168.206.53@tcp [ 1910.183865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.257864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1910.301459] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1917.527247] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 05:59:46 (1790762386) [ 1919.821693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1921.426524] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 05:59:50 (1790762390) [ 1922.185549] LNet: 103293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.193617] LNetError: 103293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.214995] LNet: Removed LNI 192.168.206.53@tcp [ 1923.023098] Key type .llcrypt unregistered [ 1923.029635] Key type ._llcrypt unregistered [ 1929.483672] Key type ._llcrypt registered [ 1929.486715] Key type .llcrypt registered [ 1929.883551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1929.936842] alg: No test for adler32 (adler32-zlib) [ 1930.918040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1930.976163] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1939.272296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1944.418307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.153@tcp [ 1944.980728] LNet: 103708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.004349] LNetError: 103708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.040976] LNet: Removed LNI 192.168.206.53@tcp [ 1945.712622] Key type .llcrypt unregistered [ 1945.717418] Key type ._llcrypt unregistered [ 1958.298840] Key type ._llcrypt registered [ 1958.301347] Key type .llcrypt registered [ 1959.637061] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1961.453242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1963.148629] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 06:00:32 (1790762432) [ 1964.528612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1966.320435] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 06:00:35 (1790762435) [ 1968.189884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg653-server [ 1969.978494] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 06:00:39 (1790762439) [ 1971.922389] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg653-server [ 1973.831484] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 06:00:42 (1790762442) [ 1975.096735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1975.175688] alg: No test for adler32 (adler32-zlib) [ 1976.302745] Lustre: Lustre: Build Version: 2.17.58_40_ge80375b [ 1976.558146] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1978.377052] Key type lgssc registered [ 1979.774407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2002.677487] Key type lgssc unregistered [ 2003.088914] LNet: 106887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.117363] LNetError: 106887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.161488] LNet: Removed LNI 192.168.206.53@tcp [ 2004.246625] Key type .llcrypt unregistered [ 2004.253237] Key type ._llcrypt unregistered [ 2020.966819] Key type ._llcrypt registered [ 2020.969534] Key type .llcrypt registered [ 2022.503681] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 06:01:31 (1790762491) [ 2022.998534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.008463] alg: No test for adler32 (adler32-zlib) [ 2024.009824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.083488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2024.126852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2024.183085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2024.242818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2031.989658] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 06:01:40 (1790762500) [ 2032.269245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.310814] LNet: 108299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.336534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.440873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2032.478556] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2033.483198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2033.531243] LNet: 108444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.544300] LNetError: 108444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.599290] LNet: Removed LNI 192.168.206.53@tcp [ 2033.621920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.712881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2033.751251] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2034.455867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2034.492790] LNetError: 108472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.518035] LNet: Removed LNI 192.168.206.53@tcp [ 2034.549519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.652184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2034.687720] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2041.289251] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 06:01:50 (1790762510) [ 2041.416770] LNet: 109053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.424284] LNet: 109053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2041.438180] LNetError: 109053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.469148] LNet: Removed LNI 192.168.206.53@tcp [ 2041.966957] Key type .llcrypt unregistered [ 2041.969178] Key type ._llcrypt unregistered [ 2048.447651] Key type ._llcrypt registered [ 2048.452910] Key type .llcrypt registered [ 2048.759514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2048.775178] alg: No test for adler32 (adler32-zlib) [ 2049.688237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2049.813423] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2049.815835] LNet: *** cfs_fail_loc=e003, val=100*** [ 2050.530668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2050.547907] LNet: *** cfs_fail_loc=e003, val=100*** [ 2050.551583] LNet: Skipped 2 previous similar messages [ 2050.558967] Ping reply received with mapping: [ 2050.562799] found metadata reply NI[192.168.206.53@tcp] [ 2050.566582] 35 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2050.569906] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2050.573543] 5a 5a 5a 5a 5a 5a 5a 5a [ 2050.759290] LNet: 109456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.766547] LNetError: 109456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.785427] LNet: Removed LNI 192.168.206.53@tcp [ 2051.417834] Key type .llcrypt unregistered [ 2051.421273] Key type ._llcrypt unregistered [ 2064.507731] Key type ._llcrypt registered [ 2064.509827] Key type .llcrypt registered [ 2066.394457] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:02:14 (1790762534) [ 2067.129658] Key type .llcrypt unregistered [ 2067.131068] Key type ._llcrypt unregistered [ 2074.610487] Key type ._llcrypt registered [ 2074.613199] Key type .llcrypt registered [ 2074.899736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2074.921629] alg: No test for adler32 (adler32-zlib) [ 2076.029737] LNetError: 110494:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2076.524493] Key type .llcrypt unregistered [ 2076.530381] Key type ._llcrypt unregistered [ 2081.805550] Key type ._llcrypt registered [ 2081.808418] Key type .llcrypt registered [ 2083.529702] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 06:02:32 (1790762552) [ 2084.400387] Key type .llcrypt unregistered [ 2084.404329] Key type ._llcrypt unregistered [ 2091.850563] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2091.856771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2094.455577] Key type ._llcrypt registered [ 2094.458745] Key type .llcrypt registered [ 2094.635742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2094.646772] alg: No test for adler32 (adler32-zlib) [ 2095.640364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.697515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2104.611525] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 06:02:53 (1790762573) [ 2104.712275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.735925] LNet: 112266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.745128] LNetError: 112266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2104.770644] LNet: Removed LNI 10.1.2.3@tcp [ 2104.800256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.489522] Key type .llcrypt unregistered [ 2105.491501] Key type ._llcrypt unregistered [ 2110.737877] Key type ._llcrypt registered [ 2110.742105] Key type .llcrypt registered [ 2110.960988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2110.979571] alg: No test for adler32 (adler32-zlib) [ 2111.890878] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2111.945298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2112.064939] LNet: 112553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.079686] LNetError: 112553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.099068] LNet: Removed LNI 192.168.206.53@tcp [ 2112.713478] Key type .llcrypt unregistered [ 2112.722928] Key type ._llcrypt unregistered [ 2124.338117] Key type ._llcrypt registered [ 2124.341690] Key type .llcrypt registered [ 2126.109991] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:03:14 (1790762594) [ 2126.887302] Key type .llcrypt unregistered [ 2126.893269] Key type ._llcrypt unregistered [ 2133.734729] Key type ._llcrypt registered [ 2133.737163] Key type .llcrypt registered [ 2133.960242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2133.981463] alg: No test for adler32 (adler32-zlib) [ 2140.900274] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:03:30 (1790762610) [ 2141.133859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.170837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.239373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2141.260138] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2141.848936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2141.870331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2141.904148] LNet: Removed LNI 192.168.206.53@tcp [ 2142.167626] LNet: 114292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.664820] Key type .llcrypt unregistered [ 2142.668411] Key type ._llcrypt unregistered [ 2153.562278] Key type ._llcrypt registered [ 2153.564955] Key type .llcrypt registered [ 2154.616287] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:03:43 (1790762623) [ 2155.167115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.194574] alg: No test for adler32 (adler32-zlib) [ 2156.114962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.189246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2156.221388] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2156.888554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2156.922840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.53@tcp [ 2156.947276] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2156.951489] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2156.954687] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2156, 0, 0 [ 2156.959727] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2156.961487] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2156.968070] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2156.971453] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2156, 0, 0 [ 2156.977614] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2156.981164] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2156.984488] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2156.987292] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2156, 0, 2206 [ 2156.993775] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2156.996806] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2157.001138] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2157.004670] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2156, 0, 2206 [ 2157.011660] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2164.003289] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:03:52 (1790762632) [ 2164.473120] LNet: 115829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.481147] LNetError: 115829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.504936] LNet: Removed LNI 192.168.206.53@tcp [ 2165.027962] Key type .llcrypt unregistered [ 2165.032030] Key type ._llcrypt unregistered [ 2171.103922] Key type ._llcrypt registered [ 2171.107806] Key type .llcrypt registered [ 2171.304929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2171.325705] alg: No test for adler32 (adler32-zlib) [ 2172.330353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2172.384569] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2180.520987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 2189.248119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.337657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2189.382706] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 2190.434875] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2190.615150] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2190.617426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2190.629573] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2193.027414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2193.058841] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2203.216927] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:04:32 (1790762672) [ 2203.540041] LNet: 117333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.548531] LNetError: 117333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.565855] LNet: Removed LNI 192.168.206.53@tcp [ 2203.578815] LNet: Removed LNI 192.168.206.53@tcp1 [ 2203.595186] LNet: Removed LNI 10.1.2.3@tcp99 [ 2204.233693] Key type .llcrypt unregistered [ 2204.236092] Key type ._llcrypt unregistered [ 2210.758333] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2210.769193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2213.487621] Key type ._llcrypt registered [ 2213.492908] Key type .llcrypt registered [ 2213.809173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2213.833903] alg: No test for adler32 (adler32-zlib) [ 2214.778381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.816367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2214.855101] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2215.564025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2215.587717] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2217.584224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2217.622789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2217.656571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2217.679787] LNetError: 118077:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2217.716815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2217.748573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2218.070135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2218.134506] LNet: 118097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.139804] LNetError: 118097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2218.146592] LNet: Removed LNI 10.1.2.3@tcp [ 2219.177973] LNet: Removed LNI 192.168.206.53@tcp [ 2219.939908] Key type .llcrypt unregistered [ 2219.943459] Key type ._llcrypt unregistered [ 2232.524664] Key type ._llcrypt registered [ 2232.528774] Key type .llcrypt registered [ 2233.540359] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:05:02 (1790762702) [ 2234.016960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2234.042219] alg: No test for adler32 (adler32-zlib) [ 2235.053497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.123090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.148585] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2235.922943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2235.981526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg653-client@tcp [ 2242.148569] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:05:11 (1790762711) [ 2242.216382] LNet: 119620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.226956] LNetError: 119620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.248400] LNet: Removed LNI 192.168.206.53@tcp [ 2242.817596] Key type .llcrypt unregistered [ 2242.820736] Key type ._llcrypt unregistered [ 2249.675801] Key type ._llcrypt registered [ 2249.680407] Key type .llcrypt registered [ 2249.893316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2249.916783] alg: No test for adler32 (adler32-zlib) [ 2250.815787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2250.844289] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2250.875372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2256.777282] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:05:26 (1790762726) [ 2258.092310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2259.811460] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:05:29 (1790762729) [ 2261.282736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2262.817576] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 06:05:32 (1790762732) [ 2262.940932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2262.967935] LNet: 121045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.974671] LNetError: 121045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.988191] LNet: Removed LNI 192.168.206.53@tcp [ 2263.052695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.117232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2263.139853] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2263.672773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2263.713970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2270.014527] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:05:39 (1790762739) [ 2270.200818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.239725] LNet: 121756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.258419] LNetError: 121756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.294061] LNet: Removed LNI 192.168.206.53@tcp [ 2270.318708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.395763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2270.442737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2271.152498] Key type .llcrypt unregistered [ 2271.156957] Key type ._llcrypt unregistered [ 2284.721449] Key type ._llcrypt registered [ 2284.724376] Key type .llcrypt registered [ 2286.172414] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:05:54 (1790762754) [ 2286.797716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2286.850335] alg: No test for adler32 (adler32-zlib) [ 2287.836267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.904740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2287.943860] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2288.926740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2289.038456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2295.545712] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:06:04 (1790762764) [ 2295.657341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.686682] LNet: 123309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.695123] LNetError: 123309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2295.727514] LNet: Removed LNI 192.168.206.53@tcp [ 2295.775505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.818953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2295.844946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2302.686977] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 06:06:11 (1790762771) [ 2302.827109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.878060] LNet: 123892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.939823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.024980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2303.423816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2303.484948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2303.539625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2303.981690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2304.024700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2304.104544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.130875] LNet: 123929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.171903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.244398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2304.632996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2304.677585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2304.797544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2305.285326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2305.346812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2311.607366] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 06:06:20 (1790762780) [ 2311.852441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.886759] LNet: 124527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.932377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.001885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2312.028565] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2313.041422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2313.069718] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2314.128530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2314.206719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2314.443638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2314.466779] LNetError: 124809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.488144] LNet: Removed LNI 192.168.206.53@tcp [ 2315.490523] LNetError: 124809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.508458] LNet: Removed LNI 192.168.206.53@tcp2 [ 2315.542891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2315.614873] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2315.624962] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2315.631384] LNet: 124815:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2315.994219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2316.176306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.194540] LNet: 124852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.202332] LNet: 124852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2317.216378] LNet: Removed LNI 192.168.206.53@tcp [ 2317.239130] LNet: Removed LNI 192.168.206.53@tcp2 [ 2317.256758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.308890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2317.332430] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2317.875343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2317.893518] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2318.509426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2318.535392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2318.575339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2318.722350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2318.737730] LNetError: 125137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.742455] LNetError: 125137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2319.776204] LNet: Removed LNI 192.168.206.53@tcp [ 2320.806399] LNet: Removed LNI 192.168.206.53@tcp2 [ 2320.835721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.908827] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2320.913840] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2320.921450] LNet: 125144:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2321.335557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2321.546861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.566198] LNet: Removed LNI 192.168.206.53@tcp [ 2322.599489] LNet: Removed LNI 192.168.206.53@tcp2 [ 2322.671546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.726234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2322.746308] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2323.324474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2323.339024] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2323.993909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2324.043517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2324.200839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.216207] LNet: 125460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.222159] LNet: 125460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2324.227764] LNetError: 125460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.238819] LNetError: 125460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2324.249698] LNet: Removed LNI 192.168.206.53@tcp [ 2324.262680] LNet: Removed LNI 192.168.206.53@tcp2 [ 2324.280078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.342143] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2324.356237] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2324.362458] LNet: 125466:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2324.797700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2324.999578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.048233] LNet: Removed LNI 192.168.206.53@tcp [ 2326.068686] LNet: Removed LNI 192.168.206.53@tcp2 [ 2326.142175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.200676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.226376] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2326.919290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2326.959153] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2328.034312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2328.089550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2328.143810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2328.355764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.440250] LNet: Removed LNI 192.168.206.53@tcp [ 2330.471350] LNet: Removed LNI 192.168.206.53@tcp2 [ 2330.488246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.569798] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2330.575165] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2330.580810] LNet: 125795:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2331.204493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2331.445212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.471872] LNet: Removed LNI 192.168.206.53@tcp [ 2332.519153] LNet: Removed LNI 192.168.206.53@tcp2 [ 2332.561191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.657354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2332.700646] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2333.486185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2333.510364] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2334.448241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2334.475633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2334.538524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2334.960952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.987862] LNetError: 126128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.994794] LNetError: 126128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2335.001630] LNet: Removed LNI 192.168.206.53@tcp [ 2335.018319] LNet: Removed LNI 192.168.206.53@tcp2 [ 2335.043549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.097479] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2335.104671] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2335.109728] LNet: 126134:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2335.610787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2335.934901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.992181] LNet: Removed LNI 192.168.206.53@tcp [ 2337.000843] LNet: Removed LNI 192.168.206.53@tcp2 [ 2337.044422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.150354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2337.207801] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2337.908322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2337.933586] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2338.776687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2338.840716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2338.887678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2338.959996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2339.302727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.384233] LNet: Removed LNI 192.168.206.53@tcp [ 2341.419270] LNet: Removed LNI 192.168.206.53@tcp2 [ 2341.451773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.521187] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2341.526394] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2342.048051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2348.497294] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2350.161150] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 06:06:59 (1790762819) [ 2350.264518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2350.286078] LNet: 127248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2350.303279] LNet: 127248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2350.312636] LNet: Removed LNI 192.168.206.53@tcp [ 2351.337628] LNet: Removed LNI 192.168.206.53@tcp2 [ 2351.395178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.649061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2351.679159] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2352.336902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2352.356710] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2353.128733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2353.152294] LNetError: 127526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2353.157882] LNetError: 127526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2353.165070] LNet: Removed LNI 10.1.2.3@tcp [ 2353.174803] LNet: Removed LNI 192.168.206.53@tcp [ 2353.204573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.290544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2353.313768] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2353.324140] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2353.774475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2353.805865] LNet: Removed LNI 192.168.206.53@tcp [ 2353.845786] LNet: Removed LNI 10.1.2.3@tcp [ 2360.035518] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 06:07:09 (1790762829) [ 2360.309780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2360.358876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.418598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.449100] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2361.081937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2361.139949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2361.193964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2361.227991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2361.264725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2361.290204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2361.319342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2361.347858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2361.380612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2361.419400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2361.453463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2361.486916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2361.597890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2368.170865] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 06:07:17 (1790762837) [ 2368.272106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.384835] LNet: Removed LNI 192.168.206.53@tcp [ 2369.440380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.562865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2369.602597] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2370.417502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2370.461074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2370.510315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2370.541381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2370.577231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2370.623895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2370.671936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2370.713510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2370.745066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2370.776858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2370.829414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2370.886959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2370.989888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2371.050295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2371.106587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2371.161636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2371.198561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2371.233919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2371.268250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2371.313563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2371.341888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2371.371858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2371.413571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2377.894659] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:07:26 (1790762846) [ 2377.998786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2379.120804] LNet: Removed LNI 192.168.206.53@tcp [ 2379.156816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2379.321565] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2379.509764] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2379.516084] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2379.531539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2381.825426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2381.846597] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2382.597497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2382.618964] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2384.731864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2384.798985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2384.879243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.53@tcp -m PUT -e local_timeout -r 1 [ 2384.955840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2385.022025] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2385.024581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2385.242511] LNetError: 129688:0:(lib-move.c:4794:lnet_parse()) 192.168.206.53@tcp, src 192.168.206.53@tcp: Dropping PUT (error -108 looking up sender) [ 2386.272525] LNet: 130146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.276786] LNet: 130146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2386.288571] LNetError: 130146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2386.299087] LNetError: 130146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2386.309969] LNet: Removed LNI 10.1.2.3@tcp [ 2387.367804] LNet: Removed LNI 192.168.206.53@tcp [ 2387.793939] Key type .llcrypt unregistered [ 2387.798602] Key type ._llcrypt unregistered [ 2398.491907] Key type ._llcrypt registered [ 2398.494477] Key type .llcrypt registered [ 2399.735666] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:07:48 (1790762868) [ 2400.133951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2400.145617] alg: No test for adler32 (adler32-zlib) [ 2401.094412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.613639] LNet: 130996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2402.311509] Key type .llcrypt unregistered [ 2402.314629] Key type ._llcrypt unregistered [ 2407.944470] Key type ._llcrypt registered [ 2407.946461] Key type .llcrypt registered [ 2408.135139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2408.158197] alg: No test for adler32 (adler32-zlib) [ 2409.089222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2409.125966] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2415.507208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 2415.686346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2416.266423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.153@tcp [ 2416.465760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2416.491972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2416.528251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2416.560550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2416.586727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2416.618536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2416.645986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2416.676351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2416.719039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2416.744368] LNet: 131444:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2416.766679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2416.801543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2416.832313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2416.864654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2416.915518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2416.947818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2417.421929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2417.651406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2417.677707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2417.703837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2417.741488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2417.797471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2417.848351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2417.878692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2417.904803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2417.949620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2417.979257] LNet: 131563:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2418.016520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2418.060963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2418.106588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2418.136574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2418.169936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2418.198825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2418.697866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.153@tcp [ 2418.864788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2418.891493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2418.930495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2418.970273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2419.004841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2419.029673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2419.053450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2419.079785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2419.106849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2419.130424] LNet: 131683:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2419.150766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2419.196656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2419.220219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2419.240945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2419.262914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2419.288545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2419.751502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2419.963767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2420.000864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2420.033328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2420.063501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2420.107411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2420.130896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2420.177822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2420.208762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2420.237688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2420.260935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2420.280338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2420.309893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2420.338882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2420.385858] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.153@tcp [ 2420.386406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2420.429879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2420.702757] LNet: 131848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.707611] LNetError: 131848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.723535] LNet: Removed LNI 192.168.206.53@tcp [ 2421.100093] Key type .llcrypt unregistered [ 2421.102265] Key type ._llcrypt unregistered [ 2435.370519] Key type ._llcrypt registered [ 2435.372630] Key type .llcrypt registered [ 2436.539823] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 06:08:25 (1790762905) [ 2436.958758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2436.976624] alg: No test for adler32 (adler32-zlib) [ 2437.952133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.517811] LNet: 132731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.276502] Key type .llcrypt unregistered [ 2439.281981] Key type ._llcrypt unregistered [ 2447.767215] Key type ._llcrypt registered [ 2447.776737] Key type .llcrypt registered [ 2448.112311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2448.130330] alg: No test for adler32 (adler32-zlib) [ 2449.161989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2449.216897] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2457.676430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 2466.278521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2466.344841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2466.397265] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 2467.585911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2468.268474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.153@tcp [ 2468.519559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2468.565739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2468.613157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2468.654988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2468.704469] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.153@tcp [ 2468.720274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2468.755491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2468.779292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2468.806708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2468.836676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2468.860575] LNet: 133383:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2468.886347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2468.910899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2468.931477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2468.960174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2468.986721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2469.021346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2469.643883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2469.953404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2469.978759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2470.007421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2470.033211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2470.059421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2470.086348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2470.112191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2470.139989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2470.181967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2470.201799] LNet: 133502:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2470.226712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2470.266281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2470.328399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2470.354609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2470.378710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2470.438077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2471.157523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.153@tcp [ 2471.484084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2471.550227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2471.620155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2471.641473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2471.668408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2471.712378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2471.761979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2471.816965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2471.845481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2471.869493] LNet: 133621:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2471.901098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2471.947524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2471.993472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2472.046147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2472.098042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2472.154982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2472.719470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2472.981458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2473.038337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2473.115574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2473.151499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2473.191061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2473.233508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2473.265714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2473.292830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2473.323542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2473.355697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2473.383568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2473.428418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2473.471694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2473.542325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2473.576456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2479.619503] LNet: 133821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2479.630122] LNetError: 133821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2479.638115] LNet: Removed LNI 192.168.206.53@tcp [ 2479.668159] LNet: Removed LNI 192.168.206.53@tcp1 [ 2480.128868] Key type .llcrypt unregistered [ 2480.130195] Key type ._llcrypt unregistered [ 2497.748842] Key type ._llcrypt registered [ 2497.750953] Key type .llcrypt registered [ 2499.415969] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 06:09:28 (1790762968) [ 2499.785758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2499.807989] alg: No test for adler32 (adler32-zlib) [ 2500.738601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2500.827503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2500.875556] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2501.761614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2501.782356] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2502.690558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2502.801210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2503.843196] LNet: Removed LNI 192.168.206.53@tcp [ 2511.053655] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 06:09:40 (1790762980) [ 2512.645896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2514.543821] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 06:09:43 (1790762983) [ 2523.215845] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 06:09:52 (1790762992) [ 2529.169821] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 06:09:58 (1790762998) [ 2536.064158] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 06:10:05 (1790763005) [ 2541.773883] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 06:10:11 (1790763011) [ 2547.533779] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 06:10:16 (1790763016) [ 2553.330468] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 06:10:22 (1790763022) [ 2559.625965] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 06:10:28 (1790763028) [ 2565.922537] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 06:10:35 (1790763035) [ 2571.652912] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 06:10:41 (1790763041) [ 2577.253824] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 06:10:46 (1790763046) [ 2583.269632] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 06:10:52 (1790763052) [ 2590.557082] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 06:10:59 (1790763059) [ 2598.033748] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 06:11:07 (1790763067) [ 2605.612232] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 06:11:14 (1790763074) [ 2612.120452] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 06:11:21 (1790763081) [ 2618.488367] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 06:11:27 (1790763087) [ 2625.880546] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 06:11:34 (1790763094) [ 2625.955530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2625.980599] LNet: 145505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2625.986948] LNetError: 145505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.002782] LNet: Removed LNI 192.168.206.53@tcp2 [ 2626.027251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2626.158294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2626.178794] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2627.394625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2627.419394] LNet: 145654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2627.428453] LNetError: 145654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2627.441682] LNet: Removed LNI 192.168.206.53@tcp [ 2627.477157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2627.535503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2627.564965] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2634.331210] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 06:11:43 (1790763103) [ 2641.869614] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 06:11:50 (1790763110) [ 2648.985940] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 06:11:58 (1790763118) [ 2656.003282] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 06:12:05 (1790763125) [ 2661.955874] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 06:12:11 (1790763131) [ 2667.959526] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 06:12:17 (1790763137) [ 2674.102845] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 06:12:23 (1790763143) [ 2680.973361] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 06:12:30 (1790763150) [ 2688.599537] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 06:12:37 (1790763157) [ 2695.658423] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 06:12:44 (1790763164) [ 2701.826313] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 06:12:51 (1790763171) [ 2708.988445] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 06:12:57 (1790763177) [ 2716.621693] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2502 sec ========== 06:13:05 (1790763185) [ 2717.186365] LNet: 153120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2717.194183] LNetError: 153120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2717.237862] LNet: Removed LNI 192.168.206.53@tcp [ 2718.068821] Key type .llcrypt unregistered [ 2718.073726] Key type ._llcrypt unregistered [ 2731.360930] Key type ._llcrypt registered [ 2731.363254] Key type .llcrypt registered [ 2731.655756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2731.680713] alg: No test for adler32 (adler32-zlib) [ 2733.039278] Lustre: Lustre: Build Version: 2.17.58_40_ge80375b [ 2733.383223] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2735.184371] Key type lgssc registered [ 2736.582771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2797.264253] Lustre: Mounted lustre-client - version 2.17.58_40_ge80375b [ 2802.632241] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2811.295200] Lustre: Unmounted lustre-client [ 2848.979798] Key type lgssc unregistered [ 2849.251264] LNet: 156066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2849.260434] LNetError: 156066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2849.277827] LNet: Removed LNI 192.168.206.53@tcp [ 2849.998309] Key type .llcrypt unregistered [ 2850.003444] Key type ._llcrypt unregistered