[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 455889553 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003266] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010172] LSM: Security Framework initializing [ 0.011076] Yama: becoming mindful. [ 0.013016] SELinux: Initializing. [ 0.014081] *** VALIDATE selinux *** [ 0.023207] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028770] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030111] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032153] *** VALIDATE tmpfs *** [ 0.034492] *** VALIDATE proc *** [ 0.036050] *** VALIDATE cgroup *** [ 0.037019] *** VALIDATE cgroup2 *** [ 0.039078] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040195] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042039] Spectre V2 : User space: Vulnerable [ 0.043011] Speculative Store Bypass: Vulnerable [ 0.046810] debug: unmapping init [mem 0xffffffff8d459000-0xffffffff8d460fff] [ 0.049197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050779] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051027] ... version: 2 [ 0.052018] ... bit width: 48 [ 0.053016] ... generic registers: 4 [ 0.054017] ... value mask: 0000ffffffffffff [ 0.055017] ... max period: 00007fffffffffff [ 0.056018] ... fixed-purpose events: 3 [ 0.057014] ... event mask: 000000070000000f [ 0.058414] rcu: Hierarchical SRCU implementation. [ 0.060714] smp: Bringing up secondary CPUs ... [ 0.061665] x86: Booting SMP configuration: [ 0.062040] .... node #0, CPUs: #1 #2 #3 [ 0.066015] smp: Brought up 1 node, 4 CPUs [ 0.068022] smpboot: Max logical packages: 1 [ 0.069021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139836] node 0 deferred pages initialised in 67ms [ 0.143287] devtmpfs: initialized [ 0.144173] x86/mm: Memory block size: 128MB [ 0.146930] gcov: version magic: 0x41383552 [ 0.148250] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150304] pinctrl core: initialized pinctrl subsystem [ 0.151251] [ 0.151896] ************************************************************* [ 0.152020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153013] ** ** [ 0.154021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155019] ** ** [ 0.156020] ** This means that this kernel is built to expose internal ** [ 0.157016] ** IOMMU data structures, which may compromise security on ** [ 0.158023] ** your system. ** [ 0.159019] ** ** [ 0.160017] ** If you see this message and you are not debugging the ** [ 0.161018] ** kernel, report this immediately to your vendor! ** [ 0.162022] ** ** [ 0.163015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164016] ************************************************************* [ 0.165767] NET: Registered protocol family 16 [ 0.166515] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169598] cpuidle: using governor menu [ 0.171969] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175559] PCI: Using configuration type 1 for base access [ 0.178158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187196] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193162] cryptd: max_cpu_qlen set to 1000 [ 0.196262] ACPI: Added _OSI(Module Device) [ 0.198023] ACPI: Added _OSI(Processor Device) [ 0.200038] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207196] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213606] ACPI: Interpreter enabled [ 0.216075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.217016] ACPI: Using IOAPIC for interrupt routing [ 0.219129] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223475] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233552] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.238021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.242099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.247410] acpiphp: Slot [2] registered [ 0.249138] acpiphp: Slot [5] registered [ 0.251124] acpiphp: Slot [6] registered [ 0.252163] acpiphp: Slot [3] registered [ 0.253093] acpiphp: Slot [4] registered [ 0.254113] acpiphp: Slot [7] registered [ 0.256101] acpiphp: Slot [8] registered [ 0.257147] acpiphp: Slot [9] registered [ 0.258100] acpiphp: Slot [10] registered [ 0.260159] acpiphp: Slot [11] registered [ 0.262127] acpiphp: Slot [12] registered [ 0.263124] acpiphp: Slot [13] registered [ 0.265133] acpiphp: Slot [14] registered [ 0.267167] acpiphp: Slot [15] registered [ 0.268123] acpiphp: Slot [16] registered [ 0.269088] acpiphp: Slot [17] registered [ 0.271126] acpiphp: Slot [18] registered [ 0.273179] acpiphp: Slot [19] registered [ 0.274113] acpiphp: Slot [20] registered [ 0.276128] acpiphp: Slot [21] registered [ 0.278168] acpiphp: Slot [22] registered [ 0.279198] acpiphp: Slot [23] registered [ 0.280102] acpiphp: Slot [24] registered [ 0.282113] acpiphp: Slot [25] registered [ 0.283146] acpiphp: Slot [26] registered [ 0.284120] acpiphp: Slot [27] registered [ 0.286132] acpiphp: Slot [28] registered [ 0.287145] acpiphp: Slot [29] registered [ 0.289122] acpiphp: Slot [30] registered [ 0.290000] acpiphp: Slot [31] registered [ 0.290000] PCI host bridge to bus 0000:00 [ 0.292031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.297028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.302025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.304035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.306257] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309085] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.312511] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319507] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.323082] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.326024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.328022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.330026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.331670] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.335173] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.337053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.339843] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.343927] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.354010] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.357019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.361991] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.373000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.383022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.401020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.411479] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.417021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.424021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.436021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.445898] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.448497] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.451483] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.454400] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.456458] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.462128] iommu: Default domain type: Passthrough [ 0.463569] SCSI subsystem initialized [ 0.466198] ACPI: bus type USB registered [ 0.467145] usbcore: registered new interface driver usbfs [ 0.470140] usbcore: registered new interface driver hub [ 0.472113] usbcore: registered new device driver usb [ 0.474224] pps_core: LinuxPPS API ver. 1 registered [ 0.476016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.479085] PTP clock support registered [ 0.482086] EDAC MC: Ver: 3.0.0 [ 0.484147] PCI: Using ACPI for IRQ routing [ 0.485695] NetLabel: Initializing [ 0.487025] NetLabel: domain hash size = 128 [ 0.488015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.490109] NetLabel: unlabeled traffic allowed by default [ 0.493091] vgaarb: loaded [ 0.494366] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.496019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.502401] clocksource: Switched to clocksource kvm-clock [ 0.609680] VFS: Disk quotas dquot_6.6.0 [ 0.611708] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.614815] *** VALIDATE ramfs *** [ 0.616403] *** VALIDATE hugetlbfs *** [ 0.619144] pnp: PnP ACPI init [ 0.621835] pnp: PnP ACPI: found 6 devices [ 0.642397] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.645793] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.648211] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.650727] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.653446] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.656955] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.660496] NET: Registered protocol family 2 [ 0.663678] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.670816] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.674607] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.680088] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.683326] TCP: Hash tables configured (established 65536 bind 65536) [ 0.686356] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.689204] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.691895] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.694706] NET: Registered protocol family 1 [ 0.697546] RPC: Registered named UNIX socket transport module. [ 0.700066] RPC: Registered udp transport module. [ 0.701997] RPC: Registered tcp transport module. [ 0.703902] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.706487] NET: Registered protocol family 44 [ 0.708381] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.710859] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.713302] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.715887] PCI: CLS 0 bytes, default 64 [ 0.717793] Unpacking initramfs... [ 2.160495] debug: unmapping init [mem 0xffff9bedfcc64000-0xffff9bedfffcffff] [ 2.164937] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.167478] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.170831] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.662749] Initialise system trusted keyrings [ 2.664638] Key type blacklist registered [ 2.666876] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.676876] zbud: loaded [ 2.680202] *** VALIDATE nfs *** [ 2.681601] *** VALIDATE nfs4 *** [ 2.683352] pstore: using deflate compression [ 2.687708] Platform Keyring initialized [ 2.795489] NET: Registered protocol family 38 [ 2.797263] Key type asymmetric registered [ 2.798872] Asymmetric key parser 'x509' registered [ 2.800962] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.804568] io scheduler mq-deadline registered [ 2.806528] io scheduler kyber registered [ 2.808381] io scheduler bfq registered [ 2.811069] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.813939] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.817722] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.821258] ACPI: Power Button [PWRF] [ 2.827321] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.834186] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.847311] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.875920] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.906232] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.912706] Non-volatile memory driver v1.3 [ 2.914473] Linux agpgart interface v0.103 [ 2.948327] virtio_blk virtio1: [vda] 146720 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.951907] vda: detected capacity change from 0 to 75120640 [ 2.970141] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.973481] vdb: detected capacity change from 0 to 1073741824 [ 2.982758] libphy: Fixed MDIO Bus: probed [ 2.988312] usbcore: registered new interface driver usbserial_generic [ 2.991289] usbserial: USB Serial support registered for generic [ 2.993715] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.998165] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.000173] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.003413] mousedev: PS/2 mouse device common for all mice [ 3.006711] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.010808] rtc_cmos 00:05: RTC can wake from S4 [ 3.014645] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.015199] rtc_cmos 00:05: registered as rtc0 [ 3.020861] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.024554] intel_pstate: CPU model not supported [ 3.027721] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.032127] hid: raw HID events driver (C) Jiri Kosina [ 3.034707] usbcore: registered new interface driver usbhid [ 3.036745] usbhid: USB HID core driver [ 3.038370] drop_monitor: Initializing network drop monitor service [ 3.040681] Initializing XFRM netlink socket [ 3.042645] NET: Registered protocol family 10 [ 3.045862] Segment Routing with IPv6 [ 3.047467] NET: Registered protocol family 17 [ 3.049133] mpls_gso: MPLS GSO support [ 3.054569] RAS: Correctable Errors collector initialized. [ 3.056499] AVX version of gcm_enc/dec engaged. [ 3.057872] AES CTR mode by8 optimization enabled [ 3.145911] sched_clock: Marking stable (3145876161, 0)->(4125626403, -979750242) [ 3.149456] registered taskstats version 1 [ 3.152063] Loading compiled-in X.509 certificates [ 3.154332] zswap: loaded using pool lzo/zbud [ 3.180950] Key type big_key registered [ 3.193288] Key type encrypted registered [ 3.194712] ima: No TPM chip found, activating TPM-bypass! [ 3.197237] ima: Allocated hash algorithm: sha1 [ 3.199078] ima: No architecture policies found [ 3.200542] evm: Initialising EVM extended attributes: [ 3.202376] evm: security.selinux [ 3.203705] evm: security.ima [ 3.205276] evm: security.capability [ 3.206691] evm: HMAC attrs: 0x1 [ 3.209153] rtc_cmos 00:05: setting system clock to 2026-09-08 04:46:31 UTC (1788842791) [ 3.215638] debug: unmapping init [mem 0xffffffff8e403000-0xffffffff8e5fffff] [ 3.219214] debug: unmapping init [mem 0xffffffff8d182000-0xffffffff8d458fff] [ 3.228136] Write protecting the kernel read-only data: 28672k [ 3.231913] debug: unmapping init [mem 0xffffffff8b803000-0xffffffff8b9fffff] [ 3.234923] debug: unmapping init [mem 0xffffffff8c114000-0xffffffff8c1fffff] [ 3.269616] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.277425] systemd[1]: Detected virtualization kvm. [ 3.278963] systemd[1]: Detected architecture x86-64. [ 3.281491] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.307162] systemd[1]: No hostname configured. [ 3.309189] systemd[1]: Set hostname to . [ 3.311773] random: systemd: uninitialized urandom read (16 bytes read) [ 3.314754] systemd[1]: Initializing machine ID from random generator. [ 3.461731] random: systemd: uninitialized urandom read (16 bytes read) [ 3.464654] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.471360] random: systemd: uninitialized urandom read (16 bytes read) [ 3.473903] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.478557] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.177536] device-mapper: uevent: version 1.0.3 [ 4.180651] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.015293] virtio_net virtio0 ens2: renamed from eth0 [ 5.076623] scsi host0: ata_piix [ 5.108804] scsi host1: ata_piix [ 5.110918] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.113298] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.506623] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.711260] random: crng init done [ 9.712551] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.195171] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.368882] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.633258] SELinux: Disabled at runtime. [ 11.695181] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.704271] systemd[1]: Detected virtualization kvm. [ 11.706337] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.232949] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.236866] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.241236] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.244989] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.250300] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.257704] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.268108] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.410493] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.790362] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.140760] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.144957] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.296881] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.309093] EDAC sbridge: Ver: 1.1.2 [ 14.529776] Key type dns_resolver registered [ 14.835885] NFS: Registering the id_resolver key type [ 14.838210] Key type id_resolver registered [ 14.839437] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg454-client login: [ 68.273024] libcfs: loading out-of-tree module taints kernel. [ 68.418809] Key type ._llcrypt registered [ 68.468044] Key type .llcrypt registered [ 69.330135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 69.349802] alg: No test for adler32 (adler32-zlib) [ 70.970479] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 71.792620] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 73.583206] Key type lgssc registered [ 75.250615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.137168] hrtimer: interrupt took 3945283 ns [ 216.784930] Lustre: Mounted lustre-client - version 2.17.57_112_g92cfe64 [ 221.953455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.545140] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing check_logdir /tmp/testlogs/ [ 241.684880] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing yml_node [ 242.658568] Lustre: lustre-OST0000-osc-ffff9bee44d5a800: disconnect after 23s idle [ 247.937985] Lustre: DEBUG MARKER: Client: 2.17.57.112 [ 250.640586] Lustre: DEBUG MARKER: MDS: 2.17.57.112 [ 253.588623] Lustre: DEBUG MARKER: OSS: 2.17.57.112 [ 255.088880] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 8 00:50:42 EDT 2026 [ 272.249130] Lustre: Unmounted lustre-client [ 299.115627] Key type lgssc unregistered [ 299.625636] LNet: 8402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.641373] LNetError: 8402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.663681] LNet: Removed LNI 192.168.204.54@tcp [ 301.200469] Key type .llcrypt unregistered [ 301.203828] Key type ._llcrypt unregistered [ 327.133346] Key type ._llcrypt registered [ 327.137638] Key type .llcrypt registered [ 327.355168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 327.377974] alg: No test for adler32 (adler32-zlib) [ 328.305866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 328.375547] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 328.401709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 333.063949] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 333.228332] LNet: 9275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.237334] LNetError: 9275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 333.270688] LNet: Removed LNI 192.168.204.54@tcp [ 333.822147] Key type .llcrypt unregistered [ 333.827669] Key type ._llcrypt unregistered [ 340.519615] Key type ._llcrypt registered [ 340.528681] Key type .llcrypt registered [ 342.000359] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:52:08 (1788843128) [ 342.384585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 342.403491] alg: No test for adler32 (adler32-zlib) [ 343.316551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.401951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.421486] LNet: 9690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.449559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.510566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 343.532211] LNet: 9697:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 350.382108] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:52:17 (1788843137) [ 350.603436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.648861] LNet: 10269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.714117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.877237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 359.284455] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:52:25 (1788843145) [ 359.391645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.443870] LNet: 10856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.476549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.583379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 366.832173] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:52:33 (1788843153) [ 366.956644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.981660] LNet: 11442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.008564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.120423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 375.096686] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:52:41 (1788843161) [ 375.327777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.368161] LNet: 12033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.388235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.461982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 376.084769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 384.268971] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:52:50 (1788843170) [ 384.364804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.387393] LNet: 12617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.417668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.485494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 384.796194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 385.276333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 392.492214] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:52:59 (1788843179) [ 392.602245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.686369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.909963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 401.255103] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:53:08 (1788843188) [ 401.349619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.407469] LNet: 13800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.413947] LNet: 13800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 401.478796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.533876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 401.554746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 401.802755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 401.862085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 402.161445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 402.217764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 402.561261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 402.593721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 403.141683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 403.193736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 403.675403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 403.715689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 404.192380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 404.214412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 404.895655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 404.935310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 405.535941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 405.596287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 406.174963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 406.222766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 406.807874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 406.844473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 407.365642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 407.434692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 408.222254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 408.275184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 417.621862] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:53:23 (1788843203) [ 417.711459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.757934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.892555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 417.975822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 426.684793] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:53:32 (1788843212) [ 426.838101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.927194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.108106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 427.171626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 437.239834] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:53:43 (1788843223) [ 437.370175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.432693] LNet: 15652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.443293] LNet: 15652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 437.491652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.651694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 437.731074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 447.854503] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:53:54 (1788843234) [ 448.032275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.129638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.254498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 448.347706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 456.129693] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:54:03 (1788843243) [ 456.414986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.493538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.583510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 456.633840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 465.022546] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:54:11 (1788843251) [ 465.153571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.207871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.321305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 465.380618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 473.270770] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:54:19 (1788843259) [ 473.337955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.367854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.492967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.793843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 480.862117] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:54:27 (1788843267) [ 480.983773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.052461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.471722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 481.842404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.170366] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:54:36 (1788843276) [ 490.561036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.627741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.839420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.072592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.981504] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:54:46 (1788843286) [ 500.440341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.503918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.106341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 501.379702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.276781] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:54:56 (1788843296) [ 509.393327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.431979] LNet: 20489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 509.435629] LNet: 20489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 509.502374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.597292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 509.665277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 517.290895] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:55:03 (1788843303) [ 517.497180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.564299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.758465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 517.804677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 526.014179] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:55:12 (1788843312) [ 526.175981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.241182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.381473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 534.963803] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:55:21 (1788843321) [ 535.155308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.235109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.348477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 544.293394] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:55:30 (1788843330) [ 544.436460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.561116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.779537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 545.490381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 553.743680] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:55:40 (1788843340) [ 553.871724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.975822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.178553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 554.282291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 563.099508] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:55:49 (1788843349) [ 563.226158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.322932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.474234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 563.504650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 571.172210] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:55:57 (1788843357) [ 571.308632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.398829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.545222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 571.579535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 579.402866] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:56:06 (1788843366) [ 579.518948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.596992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.695759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 579.778542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 587.597382] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:56:14 (1788843374) [ 587.761887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.832413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.974649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 588.127970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 588.172211] LNetError: 25810:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 588.242533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 588.286832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 595.955229] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:56:22 (1788843382) [ 596.056974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.113116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.214456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.58QnJSUCD2 [ 596.305147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.58QnJSUCD2 [ 596.372676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.58QnJSUCD2 [ 604.024633] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:56:30 (1788843390) [ 604.301553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.362142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.480448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 604.518573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 604.570092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 611.049640] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 00:56:37 (1788843397) [ 611.141966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.210487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.311828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.339692] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 613.241356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 614.849545] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 616.731427] Key type lgssc registered [ 617.998512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 621.335700] LNetError: 27570:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.54@tcp99 has no available nets [ 623.774842] Key type lgssc unregistered [ 624.204157] LNetError: 28679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.219480] LNet: Removed LNI 192.168.204.54@tcp [ 624.845410] Key type .llcrypt unregistered [ 624.850711] Key type ._llcrypt unregistered [ 631.246110] Key type ._llcrypt registered [ 631.249617] Key type .llcrypt registered [ 632.525298] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:56:59 (1788843419) [ 633.062300] Key type .llcrypt unregistered [ 633.064681] Key type ._llcrypt unregistered [ 639.511455] Key type ._llcrypt registered [ 639.521466] Key type .llcrypt registered [ 639.686137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 639.705695] alg: No test for adler32 (adler32-zlib) [ 640.588515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.646484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 640.666612] LNet: 29589:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 640.677677] LNet: 29589:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 640.683754] LNet: 29589:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 640.696182] LNet: Message forwarding will be enabled in 2s [ 643.807180] LNet: Message forwarding enabled [ 644.190761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 644.235721] LNet: Message forwarding will stop in 2s to 6s [ 647.910792] LNet: Message forwarding disabled [ 648.798522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 648.828346] LNet: 29688:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 648.833543] LNet: 29688:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 648.844100] LNet: 29688:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 648.861587] LNet: Message forwarding will be enabled in 2s [ 648.995883] LNet: 29701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.678518] Key type .llcrypt unregistered [ 649.683676] Key type ._llcrypt unregistered [ 659.192505] Key type ._llcrypt registered [ 659.194961] Key type .llcrypt registered [ 660.479798] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:57:27 (1788843447) [ 660.817192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 660.834529] alg: No test for adler32 (adler32-zlib) [ 661.852646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.947711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 668.745882] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:57:35 (1788843455) [ 669.076362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.094605] LNet: 31101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.125321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.185821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 669.218860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 669.253964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 669.292763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 669.360965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 669.400067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 669.442875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 669.506683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 669.552790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 669.602923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 669.647337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 669.700516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 669.740413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 669.779631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 669.815467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 669.876377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 669.929479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 669.963533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 670.012743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 670.046177] LNetError: 31144:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 677.338925] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:57:43 (1788843463) [ 677.457110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.502166] LNet: 31710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.545188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.652194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 684.562410] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:57:51 (1788843471) [ 686.756150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg454-server [ 689.060645] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:57:55 (1788843475) [ 689.324512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.368313] LNet: 32646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.409442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.453768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.489656] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 690.297238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[55-59]@tcp [ 690.310463] LNet: 32782:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 690.360779] LNetError: 32652:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 690.378728] LNetError: 32652:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.56@tcp has gone from up to down [ 697.227782] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:58:03 (1788843483) [ 698.923386] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg454-server [ 700.781647] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:58:07 (1788843487) [ 700.912779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.942712] LNet: 33714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.948093] LNetError: 33714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.977712] LNet: Removed LNI 192.168.204.54@tcp [ 701.046291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.133944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 701.158535] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 702.019804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[55-60/2]@tcp [ 702.051526] LNet: 33836:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 702.060921] LNet: 33836:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 702.090341] LNetError: 33722:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.59@tcp is being used as a gateway but routing feature is not turned on [ 702.109228] LNetError: 33722:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 702.117073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[55-60/2]@tcp [ 702.120528] LNetError: 33722:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.59@tcp has gone from up to down [ 702.147320] LNetError: 33722:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 709.359986] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:58:16 (1788843496) [ 709.598726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.624130] LNet: 34428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.633195] LNetError: 34428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.657038] LNet: Removed LNI 192.168.204.54@tcp [ 709.687546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.772365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 709.821785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.850793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.921870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 710.013601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.052115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.130688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 710.229799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.268458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.331140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 710.455684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.508106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.556642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 710.623289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.682612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.768413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 710.856627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.916952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.988946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 711.095702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.149992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.236587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 711.372822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.417660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.489994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 711.595462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.636507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.695572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 711.773375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.815495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.889655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 711.974851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.022950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.115793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 719.087447] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:58:25 (1788843505) [ 721.160088] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg454-server [ 722.980135] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:58:29 (1788843509) [ 725.420553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg454-server [ 727.555455] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:58:34 (1788843514) [ 727.760980] LNet: 35824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.772304] LNet: 35824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 728.563653] Key type .llcrypt unregistered [ 728.566213] Key type ._llcrypt unregistered [ 734.577193] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 734.712545] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 734.723404] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 734.732581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 737.406520] Key type ._llcrypt registered [ 737.414380] Key type .llcrypt registered [ 737.764476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 737.798928] alg: No test for adler32 (adler32-zlib) [ 738.787473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.855304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 738.890386] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 739.636148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 739.652811] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 741.854318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 741.881805] LNet: Removed LNI 10.1.2.3@tcp [ 742.031083] LNet: 36546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.039846] LNetError: 36546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.060384] LNet: Removed LNI 192.168.204.54@tcp [ 742.750822] Key type .llcrypt unregistered [ 742.752355] Key type ._llcrypt unregistered [ 753.933858] Key type ._llcrypt registered [ 753.938907] Key type .llcrypt registered [ 755.593615] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:59:01 (1788843541) [ 756.181739] Key type .llcrypt unregistered [ 756.183926] Key type ._llcrypt unregistered [ 763.794115] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 764.064554] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 764.075644] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 764.086370] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 767.315450] Key type ._llcrypt registered [ 767.317543] Key type .llcrypt registered [ 767.612569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 767.639868] alg: No test for adler32 (adler32-zlib) [ 768.694995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.763963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 768.798981] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 770.307855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 770.348944] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 773.777633] LNet: 38034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.787578] LNetError: 38034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 773.799153] LNet: Removed LNI 10.1.2.3@tcp [ 773.807658] LNet: Removed LNI 192.168.204.54@tcp [ 774.617484] Key type .llcrypt unregistered [ 774.625215] Key type ._llcrypt unregistered [ 788.368549] Key type ._llcrypt registered [ 788.371315] Key type .llcrypt registered [ 789.693832] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:59:36 (1788843576) [ 790.553532] Key type .llcrypt unregistered [ 790.555571] Key type ._llcrypt unregistered [ 797.457674] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 797.521275] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 797.523561] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 797.526037] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 800.242144] Key type ._llcrypt registered [ 800.243781] Key type .llcrypt registered [ 800.420525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.439807] alg: No test for adler32 (adler32-zlib) [ 801.374960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.434249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 801.456887] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 803.322621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 803.337897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.748895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 805.808788] LNet: Removed LNI 10.1.2.3@tcp [ 805.849756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 805.912525] LNet: Removed LNI 10.1.2.31@tcp [ 805.950798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 806.052506] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 806.847739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 806.883863] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 808.390974] LNet: 39962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.402750] LNetError: 39962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 808.405553] LNet: Removed LNI 10.1.2.31@tcp [ 808.421709] LNet: Removed LNI 10.1.2.3@tcp [ 809.057727] Key type .llcrypt unregistered [ 809.060975] Key type ._llcrypt unregistered [ 821.023085] Key type ._llcrypt registered [ 821.025801] Key type .llcrypt registered [ 822.347719] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:00:09 (1788843609) [ 822.740299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.750319] alg: No test for adler32 (adler32-zlib) [ 823.687407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.735459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.756123] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 824.639786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.665133] LNet: 40911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.677788] LNetError: 40911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.692055] LNet: Removed LNI 192.168.204.54@tcp [ 824.733539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.848837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.54@tcp [ 824.876525] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 825.485798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.54@tcp [ 825.513067] LNet: Removed LNI 192.168.204.54@tcp [ 825.584946] LNet: 41046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.282077] Key type .llcrypt unregistered [ 826.286279] Key type ._llcrypt unregistered [ 838.615453] Key type ._llcrypt registered [ 838.617634] Key type .llcrypt registered [ 840.347915] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:00:26 (1788843626) [ 842.692425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg454-server [ 845.317461] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:00:31 (1788843631) [ 846.681284] Key type .llcrypt unregistered [ 846.686357] Key type ._llcrypt unregistered [ 856.059498] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 856.276883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 856.279669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 856.287338] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 858.005965] Key type ._llcrypt registered [ 858.010908] Key type .llcrypt registered [ 858.321689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.372392] alg: No test for adler32 (adler32-zlib) [ 859.262811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.322538] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 860.034741] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 862.246489] LNet: 42866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.252776] LNetError: 42866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.261541] LNet: Removed LNI 192.168.204.54@tcp1 [ 862.283793] LNet: Removed LNI 10.1.2.3@tcp2 [ 863.009450] Key type .llcrypt unregistered [ 863.011924] Key type ._llcrypt unregistered [ 875.761374] Key type ._llcrypt registered [ 875.763609] Key type .llcrypt registered [ 877.509367] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:01:03 (1788843663) [ 877.861929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.896353] alg: No test for adler32 (adler32-zlib) [ 878.965417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.034137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.052432] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 880.286180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.322411] LNet: 43845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.335403] LNetError: 43845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.359504] LNet: Removed LNI 192.168.204.54@tcp [ 880.401833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.518872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 880.548342] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 887.818784] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:01:14 (1788843674) [ 888.326847] LNet: 44436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.331522] LNetError: 44436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.350399] LNet: Removed LNI 192.168.204.54@tcp [ 889.123602] Key type .llcrypt unregistered [ 889.124965] Key type ._llcrypt unregistered [ 897.525252] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 897.816120] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 897.824993] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 897.837894] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 900.642130] Key type ._llcrypt registered [ 900.647259] Key type .llcrypt registered [ 901.006217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.022724] alg: No test for adler32 (adler32-zlib) [ 901.958197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.011207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 902.056215] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 903.043514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 903.074041] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 906.082365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.138154] LNet: 45177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.145468] LNetError: 45177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 906.155854] LNet: Removed LNI 10.1.2.3@tcp [ 906.185128] LNet: Removed LNI 192.168.204.54@tcp [ 906.224896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.354565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 906.413989] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 906.423751] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.169263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.220038] LNet: 45204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.240137] LNetError: 45204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 907.250331] LNetError: 45204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 907.257541] LNet: Removed LNI 10.1.2.3@tcp [ 907.268073] LNet: Removed LNI 192.168.204.54@tcp [ 907.296712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.354676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 907.399095] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 907.403960] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 908.159799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 908.208192] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 909.561711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.607567] LNet: 45367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.613863] LNetError: 45367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 909.619159] LNetError: 45367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 909.624649] LNet: Removed LNI 10.1.2.3@tcp [ 909.626982] LNet: Removed LNI 192.168.204.54@tcp [ 909.643443] LNet: Removed LNI 192.168.204.54@tcp2 [ 909.669425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.769803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 909.841333] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 909.850422] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.863281] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 917.986602] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:01:44 (1788843704) [ 918.316343] LNet: 45963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.341568] LNetError: 45963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.354541] LNetError: 45963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 918.369686] LNet: Removed LNI 10.1.2.3@tcp [ 918.374422] LNet: Removed LNI 192.168.204.54@tcp [ 918.387676] LNet: Removed LNI 192.168.204.54@tcp2 [ 919.135902] Key type .llcrypt unregistered [ 919.140888] Key type ._llcrypt unregistered [ 925.292461] Key type ._llcrypt registered [ 925.294418] Key type .llcrypt registered [ 925.675277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.689360] alg: No test for adler32 (adler32-zlib) [ 926.734874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.801865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 926.842243] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 927.707948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.755092] LNet: 46354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.764356] LNetError: 46354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.779675] LNet: Removed LNI 192.168.204.54@tcp [ 927.803699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.926351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 927.960925] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 928.863594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.891633] LNet: 46385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.901777] LNetError: 46385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.928327] LNet: Removed LNI 192.168.204.54@tcp [ 928.966609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.054234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 929.101512] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 929.787853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.825342] LNet: Removed LNI 192.168.204.54@tcp [ 929.843786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.939026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 929.990397] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 930.545990] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 930.773236] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 930.789392] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 930.797606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 933.179097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.205412] LNet: 46494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.211848] LNet: 46494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 933.220026] LNetError: 46494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.226877] LNetError: 46494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 933.246862] LNet: Removed LNI 192.168.204.54@tcp [ 933.278600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.348355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 933.375139] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 940.286845] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:02:07 (1788843727) [ 940.645549] LNet: 47091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.657813] LNetError: 47091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.691678] LNet: Removed LNI 192.168.204.54@tcp [ 941.388839] Key type .llcrypt unregistered [ 941.394628] Key type ._llcrypt unregistered [ 948.754106] Key type ._llcrypt registered [ 948.759654] Key type .llcrypt registered [ 949.103429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 949.131729] alg: No test for adler32 (adler32-zlib) [ 950.109538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.209655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.262760] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 951.457661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.473528] LNet: 47487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.477736] LNetError: 47487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.487644] LNet: Removed LNI 192.168.204.54@tcp [ 951.502734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.561962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 951.586749] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 959.487394] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:02:26 (1788843746) [ 959.770924] LNet: 48079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.782709] LNetError: 48079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.806551] LNet: Removed LNI 192.168.204.54@tcp [ 960.663260] Key type .llcrypt unregistered [ 960.667400] Key type ._llcrypt unregistered [ 967.334502] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 967.508332] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 967.559481] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 967.571403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 970.336579] Key type ._llcrypt registered [ 970.341061] Key type .llcrypt registered [ 970.602302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 970.627928] alg: No test for adler32 (adler32-zlib) [ 971.673394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.752237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 971.813326] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 973.032944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 973.061418] LNet: 48520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.068975] LNetError: 48520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.087490] LNet: Removed LNI 192.168.204.54@tcp [ 973.118580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.295538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 973.347205] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 981.314451] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:02:48 (1788843768) [ 981.619110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.638827] LNet: 49118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.645619] LNetError: 49118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.677865] LNet: Removed LNI 192.168.204.54@tcp [ 981.701268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.858362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 988.766203] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:02:55 (1788843775) [ 988.859669] LNet: 49693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.724833] Key type .llcrypt unregistered [ 989.728977] Key type ._llcrypt unregistered [ 996.517314] Key type ._llcrypt registered [ 996.520192] Key type .llcrypt registered [ 996.770793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.788642] alg: No test for adler32 (adler32-zlib) [ 997.758720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.848959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 997.871921] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 998.886068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.904189] LNet: 50096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.911262] LNetError: 50096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.930470] LNet: Removed LNI 192.168.204.54@tcp [ 998.959015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.048642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 999.102462] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1007.641312] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:03:14 (1788843794) [ 1007.708447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.728666] LNet: 50682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.739229] LNetError: 50682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.748611] LNet: Removed LNI 192.168.204.54@tcp [ 1007.776898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.861653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1014.811241] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:03:21 (1788843801) [ 1016.830172] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1018.966941] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:03:25 (1788843805) [ 1020.945289] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1023.357189] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:03:29 (1788843809) [ 1025.521968] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1027.736249] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:03:34 (1788843814) [ 1029.633906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1031.528356] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:03:38 (1788843818) [ 1033.138427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1034.823276] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:03:41 (1788843821) [ 1034.973884] LNet: 52738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.723818] Key type .llcrypt unregistered [ 1035.729743] Key type ._llcrypt unregistered [ 1042.360443] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1042.488093] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1042.497170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1042.505376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1045.087899] Key type ._llcrypt registered [ 1045.091355] Key type .llcrypt registered [ 1045.313438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1045.345918] alg: No test for adler32 (adler32-zlib) [ 1046.338952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.451388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1046.472528] LNet: Added LNI 192.168.204.54@tcp [8/512/0/180] [ 1046.477424] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1046.479584] LNet: 53063:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1046.655641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.676455] LNet: 53082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.683870] LNetError: 53082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.688550] LNet: Removed LNI 10.1.2.3@tcp [ 1046.692565] LNetError: 53082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.718317] LNet: Removed LNI 192.168.204.54@tcp [ 1046.759417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.867042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1046.903496] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1046.996935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.024884] LNet: 53106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.034655] LNetError: 53106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.049173] LNet: Removed LNI 192.168.204.54@tcp [ 1047.072095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.168228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.188665] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1047.201682] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1047.203826] LNet: 53114:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1047.315574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.344043] LNet: 53132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.356139] LNetError: 53132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.377103] LNet: Removed LNI 10.1.2.3@tcp [ 1047.381478] LNetError: 53132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.414052] LNet: Removed LNI 192.168.204.54@tcp [ 1047.452572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.529620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.544846] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1047.549481] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1047.551080] LNet: 53140:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1047.769581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.787332] LNet: 53166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.800395] LNetError: 53166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.805190] LNet: Removed LNI 10.1.2.3@tcp [ 1047.809657] LNetError: 53166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.825467] LNet: Removed LNI 192.168.204.54@tcp [ 1047.849284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.914908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.945587] LNet: Added LNI 192.168.204.54@tcp [8/256/0/182] [ 1047.962617] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1048.121091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1048.162580] LNet: 53194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.176039] LNetError: 53194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1048.185801] LNet: Removed LNI 10.1.2.3@tcp [ 1048.192107] LNetError: 53194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.219159] LNet: Removed LNI 192.168.204.54@tcp [ 1048.252569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1048.355865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1048.406882] LNet: Added LNI 192.168.204.54@tcp [8/256/0/182] [ 1048.414387] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1048.416416] LNet: 53202:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1048.551579] LNet: Removed LNI 10.1.2.3@tcp [ 1048.579600] LNet: Removed LNI 192.168.204.54@tcp [ 1049.649751] Key type .llcrypt unregistered [ 1049.656130] Key type ._llcrypt unregistered [ 1064.412642] Key type ._llcrypt registered [ 1064.417239] Key type .llcrypt registered [ 1066.338484] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:04:12 (1788843852) [ 1068.370705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1070.461704] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:04:17 (1788843857) [ 1070.829194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1070.846616] alg: No test for adler32 (adler32-zlib) [ 1071.812695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.855025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1071.893438] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1072.744701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.54 [ 1072.762187] LNet: 54464:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1072.773978] LNet: Removed LNI 192.168.204.54@tcp [ 1080.536525] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:04:26 (1788843866) [ 1080.665834] LNet: 55026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.685071] LNetError: 55026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.709386] LNet: Removed LNI 192.168.204.54@tcp [ 1081.632893] Key type .llcrypt unregistered [ 1081.634714] Key type ._llcrypt unregistered [ 1089.425351] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1089.764662] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1091.712906] Key type ._llcrypt registered [ 1091.715472] Key type .llcrypt registered [ 1092.073191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.097512] alg: No test for adler32 (adler32-zlib) [ 1093.718923] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1101.504159] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:04:48 (1788843888) [ 1101.787477] LNet: 55934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.811530] LNetError: 55934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1101.830592] LNet: Removed LNI 10.1.2.3@tcp [ 1102.751833] Key type .llcrypt unregistered [ 1102.753785] Key type ._llcrypt unregistered [ 1110.954801] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1111.554761] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1111.561877] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1111.572436] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1114.775775] Key type ._llcrypt registered [ 1114.777446] Key type .llcrypt registered [ 1115.134415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1115.164807] alg: No test for adler32 (adler32-zlib) [ 1116.253480] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1125.982883] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:05:12 (1788843912) [ 1127.780836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1130.462786] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:05:16 (1788843916) [ 1130.651796] LNet: 57116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.659670] LNetError: 57116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1130.696683] LNet: Removed LNI 10.1.2.3@tcp [ 1131.648652] Key type .llcrypt unregistered [ 1131.651922] Key type ._llcrypt unregistered [ 1139.990491] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1140.367600] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1140.374452] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1140.389303] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1143.480501] Key type ._llcrypt registered [ 1143.488501] Key type .llcrypt registered [ 1143.710420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1143.749344] alg: No test for adler32 (adler32-zlib) [ 1144.731547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1144.825597] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1155.285941] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:05:41 (1788843941) [ 1155.946880] LNet: 58350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.954386] LNetError: 58350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1155.975518] LNet: Removed LNI 10.1.2.3@tcp [ 1156.976758] Key type .llcrypt unregistered [ 1156.982384] Key type ._llcrypt unregistered [ 1164.023994] Key type ._llcrypt registered [ 1164.028805] Key type .llcrypt registered [ 1164.422438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1164.465717] alg: No test for adler32 (adler32-zlib) [ 1165.498915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1165.576362] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1173.520166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1174.356917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_interrupt -r 1 [ 1174.410676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_interrupt -r 1 [ 1174.448512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1174.476177] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1174.693666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.719156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.749036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.783950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.810806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.815178] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1174.838776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.870451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.891938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.909339] LNet: 58785:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.929504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.961624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.994717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.028791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.067667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.098829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.128081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1175.753571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_dropped -r 1 [ 1175.798434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_dropped -r 1 [ 1175.831267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1175.851023] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1176.098566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.126639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.186594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.246884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.316761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.415761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.472709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.523156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.571500] LNet: 58894:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.618656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.672608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.727763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.778435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.821685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.849669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.865891] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1176.946919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1177.680313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_aborted -r 1 [ 1177.732796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_aborted -r 1 [ 1177.764922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1178.061509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.088441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.116346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.139412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.162663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.196125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.233562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.274702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.297184] LNet: 59003:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.326864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.387617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.437129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.512089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.569634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.622352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.680347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.477795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_no_route -r 1 [ 1179.532508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_no_route -r 1 [ 1179.562853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1179.580559] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1179.586185] LNet: Skipped 1 previous similar message [ 1179.777948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.802973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.826792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.848796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.872287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.911735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.947988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.988661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.025576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.055279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.089307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.118719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.145615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.173960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.207511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1180.700415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_timeout -r 1 [ 1180.740461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_timeout -r 1 [ 1180.766363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1181.135858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.211197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.270994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.309577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.357689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.400826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.460632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.505602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.528500] LNet: 59222:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.533462] LNet: 59222:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1181.548457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.610225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.660238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.723279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.777373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.831820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.896442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1182.527795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_error -r 1 [ 1182.578674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_error -r 1 [ 1182.624299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1182.965267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.027906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.064617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.112295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.150203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.197724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.251255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.296434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.365737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.447627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.509291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.551746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.580160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.618599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.654976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.904470] LNet: 59372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.928849] LNetError: 59372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.948512] LNet: Removed LNI 192.168.204.54@tcp [ 1184.407477] Key type .llcrypt unregistered [ 1184.414346] Key type ._llcrypt unregistered [ 1203.584939] Key type ._llcrypt registered [ 1203.587041] Key type .llcrypt registered [ 1204.850733] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:06:31 (1788843991) [ 1206.501596] Key type .llcrypt unregistered [ 1206.504686] Key type ._llcrypt unregistered [ 1212.090654] Key type ._llcrypt registered [ 1212.096505] Key type .llcrypt registered [ 1212.366593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1212.393788] alg: No test for adler32 (adler32-zlib) [ 1213.328676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1213.349399] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1221.427842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1233.959163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1234.031372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1234.084503] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1236.048925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_interrupt -r 1 [ 1236.113507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_interrupt -r 1 [ 1236.183497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_interrupt -r 1 [ 1236.252485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_interrupt -r 1 [ 1236.333732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_interrupt -r 1 [ 1236.444233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_interrupt -r 1 [ 1236.514634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_interrupt -r 1 [ 1236.560892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_interrupt -r 1 [ 1236.629974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1236.648332] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1236.652699] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1236.863561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.893286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.920875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.950440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.980464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.007949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.037422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.071917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.090640] LNet: 60837:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.118485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.160087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.205809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.246728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.300270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.360847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.410795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1238.128108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_dropped -r 1 [ 1238.180107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_dropped -r 1 [ 1238.278406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_dropped -r 1 [ 1238.306584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_dropped -r 1 [ 1238.332690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_dropped -r 1 [ 1238.364742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_dropped -r 1 [ 1238.403514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_dropped -r 1 [ 1238.447757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_dropped -r 1 [ 1238.499505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1238.534931] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1238.541772] LNet: Skipped 2 previous similar messages [ 1238.831400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.888140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1238.939344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.973557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.003383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.061613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.137438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.191484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.224224] LNet: 60958:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.244600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.288274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.327461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.358108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.420779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.458367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.484446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1240.096157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_aborted -r 1 [ 1240.132406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_aborted -r 1 [ 1240.181109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_aborted -r 1 [ 1240.221122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_aborted -r 1 [ 1240.298895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_aborted -r 1 [ 1240.352261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_aborted -r 1 [ 1240.414646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_aborted -r 1 [ 1240.448210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_aborted -r 1 [ 1240.488636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1240.512125] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1240.518403] LNet: Skipped 2 previous similar messages [ 1240.769764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.811336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.844708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.875616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.938357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.993726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.077218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.131702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.172337] LNet: 61079:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.201244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.261520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.309149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.355553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.410984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.459838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.484299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1241.964704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_no_route -r 1 [ 1241.997809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_no_route -r 1 [ 1242.029487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_no_route -r 1 [ 1242.056416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_no_route -r 1 [ 1242.110669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_no_route -r 1 [ 1242.148721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_no_route -r 1 [ 1242.209479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_no_route -r 1 [ 1242.258708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_no_route -r 1 [ 1242.319582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1242.540343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.563819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1242.585811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1242.606113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1242.626830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1242.650808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1242.672439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1242.692734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1242.716275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1242.738507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.761371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.786819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.815761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.848489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.892530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1243.526215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_timeout -r 1 [ 1243.565237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_timeout -r 1 [ 1243.613644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_timeout -r 1 [ 1243.669131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_timeout -r 1 [ 1243.721808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_timeout -r 1 [ 1243.771471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_timeout -r 1 [ 1243.887789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_timeout -r 1 [ 1243.960808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_timeout -r 1 [ 1244.055882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1244.107419] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1244.117543] LNet: Skipped 5 previous similar messages [ 1244.413378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.454859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.489190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.517139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.555622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.596255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.632893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.688521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.721142] LNet: 61322:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.735453] LNet: 61322:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1244.759490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.795823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.838982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.877702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.934587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.977624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.019492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1245.749644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_error -r 1 [ 1245.807515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_error -r 1 [ 1245.858201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_error -r 1 [ 1245.904315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_error -r 1 [ 1245.964781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_error -r 1 [ 1246.028555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_error -r 1 [ 1246.097284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_error -r 1 [ 1246.159049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_error -r 1 [ 1246.224146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1246.631214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.716675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1246.759659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.804273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.849489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.885591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.934725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.977764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.015883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.059139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.084966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.107959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.132751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.168142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.225559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1252.916584] LNet: 61520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.925459] LNetError: 61520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.935329] LNet: Removed LNI 192.168.204.54@tcp [ 1252.951764] LNet: Removed LNI 192.168.204.54@tcp1 [ 1253.588203] Key type .llcrypt unregistered [ 1253.590493] Key type ._llcrypt unregistered [ 1272.318992] Key type ._llcrypt registered [ 1272.322779] Key type .llcrypt registered [ 1273.720501] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:07:40 (1788844060) [ 1274.859608] Key type .llcrypt unregistered [ 1274.861779] Key type ._llcrypt unregistered [ 1282.230514] Key type ._llcrypt registered [ 1282.233404] Key type .llcrypt registered [ 1282.489450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1282.501951] alg: No test for adler32 (adler32-zlib) [ 1283.519742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1283.570886] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1291.874517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1292.815312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_dropped -r 1 [ 1292.875113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_dropped -r 1 [ 1292.941751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1292.970720] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1293.229511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1293.283726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1293.316761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1293.344137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1293.393712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1293.426740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1293.463457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1293.507419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1293.537603] LNet: 62765:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1293.572891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1293.625624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1293.657158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.747530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.810497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.847662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.932197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1294.534274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_error -r 1 [ 1294.563325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_error -r 1 [ 1294.591566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1294.606826] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1294.917472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1294.969959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1295.041673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.089568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.164779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.220424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.263666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.296630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.316593] LNet: 62875:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.340406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1295.395199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1295.454792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1295.491262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1295.536707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1295.567317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1295.593341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1296.256637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_timeout -r 1 [ 1296.298509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_timeout -r 1 [ 1296.335784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1296.353731] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1296.601355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1296.658636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1296.735374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1296.798967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1296.882446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1296.976442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.052735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.102397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.126405] LNet: 62984:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.146715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.169486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.213928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.267834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.330705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.382110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.441402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1297.811518] LNet: 63025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.818968] LNetError: 63025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.840379] LNet: Removed LNI 192.168.204.54@tcp [ 1298.410512] Key type .llcrypt unregistered [ 1298.415261] Key type ._llcrypt unregistered [ 1320.218882] Key type ._llcrypt registered [ 1320.220609] Key type .llcrypt registered [ 1321.710293] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:08:28 (1788844108) [ 1322.852273] Key type .llcrypt unregistered [ 1322.855210] Key type ._llcrypt unregistered [ 1330.126992] Key type ._llcrypt registered [ 1330.134947] Key type .llcrypt registered [ 1330.581567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1330.609053] alg: No test for adler32 (adler32-zlib) [ 1331.529455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1331.565881] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1339.251259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1349.915413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.985934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1350.034801] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1352.153885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_dropped -r 1 [ 1352.224983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e remote_dropped -r 1 [ 1352.300200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_dropped -r 1 [ 1352.344589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e remote_dropped -r 1 [ 1352.395499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e remote_dropped -r 1 [ 1352.437393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e remote_dropped -r 1 [ 1352.506029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e remote_dropped -r 1 [ 1352.560844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e remote_dropped -r 1 [ 1352.629118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1352.654534] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1352.673335] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.154@tcp [ 1352.982798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1353.049629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1353.095296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1353.134447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1353.168898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1353.224720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1353.279911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1353.327665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1353.367913] LNet: 64493:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1353.386542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1353.434582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1353.463622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1353.496726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1353.535457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1353.571276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1353.610658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1353.653228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1354.204195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_error -r 1 [ 1354.242650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e remote_error -r 1 [ 1354.296932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_error -r 1 [ 1354.327851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e remote_error -r 1 [ 1354.382388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e remote_error -r 1 [ 1354.450322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e remote_error -r 1 [ 1354.503709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e remote_error -r 1 [ 1354.547594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e remote_error -r 1 [ 1354.623801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1354.675968] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1354.685144] LNet: Skipped 2 previous similar messages [ 1354.933908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1354.972874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1355.005421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1355.032409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1355.060411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1355.083450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1355.141894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1355.241546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1355.274195] LNet: 64617:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1355.311316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1355.368905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1355.426653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1355.489076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1355.531297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1355.604968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1355.672446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1355.711165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1356.486977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_timeout -r 1 [ 1356.519780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e remote_timeout -r 1 [ 1356.549882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_timeout -r 1 [ 1356.600237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e remote_timeout -r 1 [ 1356.639791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e remote_timeout -r 1 [ 1356.684907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e remote_timeout -r 1 [ 1356.723216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e remote_timeout -r 1 [ 1356.759560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e remote_timeout -r 1 [ 1356.802924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1356.838899] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1357.076849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1357.101344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1357.126603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1357.151505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1357.176277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1357.226956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1357.268193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1357.293223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1357.312348] LNet: 64740:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1357.330101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1357.387419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1357.431411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1357.488868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1357.524633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1357.579962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1357.619239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1357.660880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1363.690599] LNet: 64818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.699402] LNetError: 64818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.703618] LNet: Removed LNI 192.168.204.54@tcp [ 1363.715807] LNet: Removed LNI 192.168.204.54@tcp1 [ 1364.236749] Key type .llcrypt unregistered [ 1364.240519] Key type ._llcrypt unregistered [ 1383.138412] Key type ._llcrypt registered [ 1383.141344] Key type .llcrypt registered [ 1384.760561] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:09:31 (1788844171) [ 1386.479570] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1388.489526] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:09:35 (1788844175) [ 1389.448471] Key type .llcrypt unregistered [ 1389.450714] Key type ._llcrypt unregistered [ 1396.531256] Key type ._llcrypt registered [ 1396.533336] Key type .llcrypt registered [ 1396.747154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1396.765153] alg: No test for adler32 (adler32-zlib) [ 1397.752994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1397.794365] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1406.595662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1407.693287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e network_timeout -r 1 [ 1407.746378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e network_timeout -r 1 [ 1407.810487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1407.833381] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1408.034438] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1408.138924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1408.181815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1408.226227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1408.279557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1408.336533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1408.436940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1408.484224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1408.561221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1408.624531] LNet: 66360:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1408.684450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1408.801359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1408.869153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1408.959546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1409.031644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1409.107586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1409.601116] LNet: 66399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.629332] LNetError: 66399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.651966] LNet: Removed LNI 192.168.204.54@tcp [ 1410.395750] Key type .llcrypt unregistered [ 1410.397945] Key type ._llcrypt unregistered [ 1431.064636] Key type ._llcrypt registered [ 1431.066752] Key type .llcrypt registered [ 1431.300635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1431.317962] alg: No test for adler32 (adler32-zlib) [ 1432.279491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1432.339553] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1440.023199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1449.646413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.679813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1449.703842] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1451.000184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e network_timeout -r 1 [ 1451.036470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e network_timeout -r 1 [ 1451.074144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e network_timeout -r 1 [ 1451.110312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e network_timeout -r 1 [ 1451.147388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e network_timeout -r 1 [ 1451.219294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e network_timeout -r 1 [ 1451.286745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e network_timeout -r 1 [ 1451.340411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e network_timeout -r 1 [ 1451.419774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1451.468709] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1451.669577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1451.695662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1451.720144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1451.741481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1451.764524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1451.784948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1451.810447] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1451.815692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1451.823620] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.154@tcp [ 1451.848927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1451.861343] LNet: 67275:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1451.876483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1451.898910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1451.925316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1451.953130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1451.977782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1452.030315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1457.189525] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1457.674451] LNet: 67350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.703953] LNetError: 67350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.722372] LNet: Removed LNI 192.168.204.54@tcp [ 1458.724478] LNetError: 67350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.754977] LNet: Removed LNI 192.168.204.54@tcp1 [ 1459.424589] Key type .llcrypt unregistered [ 1459.426404] Key type ._llcrypt unregistered [ 1479.959183] Key type ._llcrypt registered [ 1479.961600] Key type .llcrypt registered [ 1481.427245] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:11:08 (1788844268) [ 1482.287610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1482.331501] alg: No test for adler32 (adler32-zlib) [ 1483.364213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.416760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.452861] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1484.268096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1484.301053] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1485.380229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 1485.516952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1485.571493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1485.608355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1485.647986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1485.717725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1485.774574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.54@tcp [ 1485.812770] LNetError: 68449:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.54@tcp to recovery [ 1486.495703] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1486.501310] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1488.543546] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1489.591411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1492.640345] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1493.210824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1493.245844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1493.302778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1493.321364] LNet: 68557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.327289] LNetError: 68557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.367445] LNet: Removed LNI 192.168.204.54@tcp [ 1494.369546] LNetError: 68557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.385393] LNet: Removed LNI 192.168.204.54@tcp1 [ 1494.403824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.499504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1494.542780] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1495.501871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1495.543579] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1496.339603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 1496.373599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1496.435746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1496.485443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1496.524525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1496.578833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1496.629044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1496.687513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.54@tcp [ 1496.713733] LNetError: 68829:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.54@tcp to recovery [ 1497.507227] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1500.420753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1503.647974] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1503.660788] LNet: Skipped 1 previous similar message [ 1508.799192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1508.875903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1508.987281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1517.603636] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:11:44 (1788844304) [ 1517.739106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.784252] LNet: 69539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.798864] LNetError: 69539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.820166] LNet: Removed LNI 192.168.204.54@tcp [ 1517.868738] LNet: Removed LNI 192.168.204.54@tcp1 [ 1517.946977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.075460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1518.150865] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1519.280375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1519.323481] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1520.506779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 1520.639879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1520.691080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1520.729780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1520.790706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1520.844903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1520.904149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.54@tcp --health 0 [ 1521.069479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1521.120636] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.54@tcp [ 1521.126107] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1521.138226] LNet: Skipped 1 previous similar message [ 1535.853958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1536.141497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1536.243432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1536.331610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.54@tcp --health 500 [ 1536.528085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1543.462522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1543.544778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.571417] LNet: 70031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.579679] LNetError: 70031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.589852] LNetError: 70031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1543.594672] LNet: Removed LNI 192.168.204.54@tcp [ 1543.611881] LNet: Removed LNI 192.168.204.54@tcp1 [ 1543.650308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.740642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.777272] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1544.491568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1544.512817] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1545.460303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 1545.506233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1545.568845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1545.598795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1545.632421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1545.671686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1545.728162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1545.781670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.54@tcp --health 0 [ 1546.783697] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1546.791360] LNet: Skipped 3 previous similar messages [ 1547.207786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1557.791070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1557.822326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1557.851079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1565.647524] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:12:32 (1788844352) [ 1566.498180] LNet: 71077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.504614] LNetError: 71077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.510799] LNetError: 71077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1566.519549] LNet: Removed LNI 192.168.204.54@tcp [ 1566.532659] LNet: Removed LNI 192.168.204.54@tcp1 [ 1567.198529] Key type .llcrypt unregistered [ 1567.204668] Key type ._llcrypt unregistered [ 1575.015961] Key type ._llcrypt registered [ 1575.019514] Key type .llcrypt registered [ 1575.306884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1575.332533] alg: No test for adler32 (adler32-zlib) [ 1576.360303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1576.413445] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1585.891617] Lustre: DEBUG MARKER: Initial discovery [ 1585.949142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.154@tcp [ 1588.696390] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1588.762204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1588.872977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.154@tcp [ 1588.903244] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1588.935439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1588.992780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.154@tcp -r 1 -m GET -l 3 [ 1589.046429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.154@tcp -r 1 -m GET -e local_error [ 1589.088639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.154@tcp -r 1 -m PUT -l 6 [ 1589.728710] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1591.076547] Lustre: DEBUG MARKER: Force oleg454-client.virtnet to discover 192.168.204.154@tcp (in background) [ 1591.129728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.154@tcp [ 1592.770430] Lustre: DEBUG MARKER: Force oleg454-server to discover 192.168.204.54@tcp [ 1594.210975] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1596.194832] Lustre: DEBUG MARKER: Wait for 71829 [ 1597.851657] Lustre: DEBUG MARKER: Finished wait on 71829 [ 1598.053474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1598.098587] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1598.116456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1598.474477] LNet: 72288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.487323] LNetError: 72288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1598.508460] LNet: Removed LNI 192.168.204.54@tcp [ 1598.972882] Key type .llcrypt unregistered [ 1598.976706] Key type ._llcrypt unregistered [ 1618.248066] Key type ._llcrypt registered [ 1618.251384] Key type .llcrypt registered [ 1620.012176] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:13:26 (1788844406) [ 1620.878256] Key type .llcrypt unregistered [ 1620.880591] Key type ._llcrypt unregistered [ 1627.849643] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1628.086475] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1628.097754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1628.111267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1630.940497] Key type ._llcrypt registered [ 1630.941875] Key type .llcrypt registered [ 1631.145608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1631.168923] alg: No test for adler32 (adler32-zlib) [ 1632.155686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.264214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1632.315546] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1632.956664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1632.973036] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1643.083098] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:13:49 (1788844429) [ 1643.419445] LNet: 74361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.440609] LNetError: 74361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1643.453465] LNet: Removed LNI 10.1.2.3@tcp [ 1643.462890] LNet: Removed LNI 192.168.204.54@tcp [ 1644.171451] Key type .llcrypt unregistered [ 1644.173505] Key type ._llcrypt unregistered [ 1652.120882] Key type ._llcrypt registered [ 1652.126263] Key type .llcrypt registered [ 1652.399904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1652.427688] alg: No test for adler32 (adler32-zlib) [ 1653.512262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.594313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1653.647823] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1654.438193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1654.455213] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1655.553260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.54@tcp [ 1656.671270] LNetError: Unexpected error -22 connecting to 192.168.204.54@tcp at host 192.168.204.54:988 [ 1663.356175] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:14:09 (1788844449) [ 1663.520359] LNet: 75483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.531222] LNetError: 75483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1663.536263] LNet: Removed LNI 10.1.2.3@tcp [ 1664.556770] LNet: Removed LNI 192.168.204.54@tcp [ 1665.338514] Key type .llcrypt unregistered [ 1665.341390] Key type ._llcrypt unregistered [ 1673.831239] Key type ._llcrypt registered [ 1673.835049] Key type .llcrypt registered [ 1674.181368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1674.217454] alg: No test for adler32 (adler32-zlib) [ 1675.181738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.240351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1675.287863] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1676.127578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1676.145965] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 1677.114819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.54@tcp1 --nid 192.168.204.54@tcp2 [ 1682.357767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp1 192.168.204.54@tcp2 [ 1682.408873] LNetError: 76323:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.54@tcp1 to 192.168.204.54@tcp2 - Route table may be misconfigured [ 1682.488085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp2 192.168.204.54@tcp1 [ 1692.767524] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:14:38 (1788844478) [ 1692.933740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.960977] LNet: 76890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.969935] LNetError: 76890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.985076] LNet: Removed LNI 192.168.204.54@tcp1 [ 1693.989492] LNetError: 76890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.009473] LNet: Removed LNI 192.168.204.54@tcp2 [ 1694.058755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.175820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.207074] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1695.318629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1695.369084] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1696.755201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 1696.828415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.54@tcp [ 1696.878417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp1 192.168.204.54@tcp1 [ 1696.923800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.54@tcp -d 192.168.204.54@tcp -e network_timeout [ 1696.960669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -e network_timeout [ 1696.996541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -e network_timeout [ 1697.036733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -e network_timeout [ 1697.075805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1697.116903] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1697.155989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1697.184465] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1697.306247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1697.434519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1705.785689] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:14:52 (1788844492) [ 1706.250646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.288249] LNet: 77759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.303054] LNetError: 77759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.311843] LNet: Removed LNI 192.168.204.54@tcp [ 1707.366281] LNet: Removed LNI 192.168.204.54@tcp1 [ 1707.506384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.718983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1708.059135] LNet: 77796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.721450] Key type .llcrypt unregistered [ 1708.731723] Key type ._llcrypt unregistered [ 1723.569283] Key type ._llcrypt registered [ 1723.575740] Key type .llcrypt registered [ 1725.351153] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:15:11 (1788844511) [ 1725.956233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1725.985862] alg: No test for adler32 (adler32-zlib) [ 1727.031568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.866853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1731.080459] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:15:17 (1788844517) [ 1731.528684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.568371] LNet: 78892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.595779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.701242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1731.758216] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1732.899735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1732.949836] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1733.911912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1733.947689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp1 [ 1733.995891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp1 [ 1742.686384] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:15:29 (1788844529) [ 1745.166650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg454-server [ 1747.124446] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:15:33 (1788844533) [ 1749.540490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg454-server [ 1751.881828] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:15:38 (1788844538) [ 1753.808673] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg454-server [ 1755.843317] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:15:42 (1788844542) [ 1757.468979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg454-server [ 1759.378082] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:15:46 (1788844546) [ 1761.233587] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg454-server [ 1762.862763] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:15:49 (1788844549) [ 1765.075688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg454-server [ 1766.977698] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:15:53 (1788844553) [ 1769.240352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg454-server [ 1771.238219] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:15:57 (1788844557) [ 1773.221625] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg454-server [ 1775.665833] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:16:02 (1788844562) [ 1777.546497] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg454-server [ 1779.471914] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:16:06 (1788844566) [ 1779.840579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.867499] LNet: 82771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.879558] LNetError: 82771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.887660] LNet: Removed LNI 192.168.204.54@tcp [ 1780.901961] LNet: Removed LNI 192.168.204.54@tcp1 [ 1780.941526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.029204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.059884] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1781.846591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1781.925287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1782.048509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.079259] LNet: 82918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.084265] LNetError: 82918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.089399] LNetError: 82918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1782.103887] LNet: Removed LNI 192.168.204.54@tcp [ 1782.130154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.216904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1782.245155] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1783.046712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1783.349143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.397092] LNetError: 83064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.416569] LNet: Removed LNI 192.168.204.54@tcp [ 1783.442836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.503350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1783.530764] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1784.116571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1784.184726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1784.379577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.423425] LNet: 83211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.430565] LNet: 83211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1785.508936] LNet: Removed LNI 192.168.204.54@tcp [ 1785.550549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.666918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1785.735203] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1786.523469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1786.689853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.707598] LNetError: 83356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.714513] LNetError: 83356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1786.728312] LNet: Removed LNI 192.168.204.54@tcp [ 1786.755755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.822419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.852173] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1787.814562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1787.941273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1788.326497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.433050] LNet: Removed LNI 192.168.204.54@tcp [ 1789.479365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.648687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1789.716425] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1790.897737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1791.217975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.251781] LNet: 83649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.256952] LNet: 83649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1791.265555] LNetError: 83649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.274064] LNetError: 83649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1791.298095] LNet: Removed LNI 192.168.204.54@tcp [ 1791.342578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.402130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1791.427502] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1792.293442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1792.392545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1792.707793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.766788] LNet: Removed LNI 192.168.204.54@tcp [ 1793.869231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.005425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1794.061499] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1795.334469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1795.713906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.786564] LNet: Removed LNI 192.168.204.54@tcp [ 1796.833822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.934799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1796.983127] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1797.857835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1797.927536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1799.453101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.484506] LNet: 84098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.501516] LNet: 84098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1799.523619] LNetError: 84098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.541401] LNetError: 84098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1799.560907] LNet: Removed LNI 192.168.204.54@tcp [ 1799.609353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.694453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1799.723939] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1801.001422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1803.713482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.776173] LNet: Removed LNI 192.168.204.54@tcp [ 1804.805167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.901648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.950478] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1806.176040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1806.323273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1808.163503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.254738] LNet: Removed LNI 192.168.204.54@tcp [ 1809.329380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.437305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1809.485358] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1810.587431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1810.960434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.019140] LNet: Removed LNI 192.168.204.54@tcp [ 1812.083419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.148678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.210172] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1813.262181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1813.353975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.411994] LNet: Removed LNI 192.168.204.54@tcp [ 1813.464302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.531457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1813.547347] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1814.237445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1814.301416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.335611] LNet: Removed LNI 192.168.204.54@tcp [ 1814.357607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.452660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1814.556894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.614161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.660256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1814.747634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.781709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.850544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1814.969981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.024305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.099821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1815.125245] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1815.957265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.991939] LNet: 85003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.001645] LNet: 85003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1816.015307] LNetError: 85003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.021056] LNetError: 85003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1816.056208] LNet: Removed LNI 192.168.204.54@tcp [ 1816.073291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.183162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1823.386136] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:16:50 (1788844610) [ 1823.585919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.681773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.781411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1823.805975] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/180] [ 1825.272228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1825.309758] LNet: Removed LNI 192.168.204.54@tcp231 [ 1825.364208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.463396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1825.508102] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/180] [ 1825.516254] LNet: 85731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1826.015645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1826.067458] LNet: Removed LNI 192.168.204.54@tcp231 [ 1826.089051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1826.121120] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/180] [ 1827.566305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.632193] LNet: Removed LNI 192.168.204.54@tcp231 [ 1827.649749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.743557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1827.765978] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/0] [ 1827.784502] LNet: 85883:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1828.199630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1828.241286] LNet: Removed LNI 192.168.204.54@tcp231 [ 1828.267690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1828.316693] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/0] [ 1829.710315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.758764] LNet: Removed LNI 192.168.204.54@tcp231 [ 1829.781903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.847974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1829.893934] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/60] [ 1829.902289] LNet: 86035:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1830.585285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1830.635217] LNet: Removed LNI 192.168.204.54@tcp231 [ 1830.678705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1830.721162] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/60] [ 1832.242558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1832.304269] LNet: Removed LNI 192.168.204.54@tcp231 [ 1832.328374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.416896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1832.443888] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/180] [ 1832.449988] LNet: 86186:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1832.910407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1832.943506] LNet: Removed LNI 192.168.204.54@tcp231 [ 1832.956744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1832.994980] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/180] [ 1841.729986] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:17:08 (1788844628) [ 1841.983697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1842.017477] LNet: Removed LNI 192.168.204.54@tcp231 [ 1842.044510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1842.113206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1842.141066] LNet: Added LNI 192.168.204.54@tcp232 [8/256/0/180] [ 1842.881699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1842.940712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1842.995070] LNet: Removed LNI 192.168.204.54@tcp232 [ 1843.036318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1843.094936] LNet: Added LNI 192.168.204.54@tcp232 [8/256/0/180] [ 1844.291655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1844.329801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1844.368513] LNet: Removed LNI 192.168.204.54@tcp232 [ 1844.405539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1844.445149] LNet: Added LNI 192.168.204.54@tcp232 [8/256/0/180] [ 1845.319607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1845.377744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1845.410440] LNet: Removed LNI 192.168.204.54@tcp232 [ 1845.426402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1845.464410] LNet: Added LNI 192.168.204.54@tcp232 [8/256/0/180] [ 1846.276897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1846.309739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1846.352420] LNet: Removed LNI 192.168.204.54@tcp232 [ 1846.370897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1854.267287] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:17:20 (1788844640) [ 1854.993516] LNet: 88024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.009829] LNet: 88024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1856.165561] Key type .llcrypt unregistered [ 1856.184146] Key type ._llcrypt unregistered [ 1864.865391] Key type ._llcrypt registered [ 1864.867080] Key type .llcrypt registered [ 1865.214483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1865.225677] alg: No test for adler32 (adler32-zlib) [ 1866.237463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1866.287618] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1876.433957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1887.277945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.337474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1887.363701] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1889.386662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_interrupt -r 2 [ 1889.434316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_interrupt -r 2 [ 1889.466106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_interrupt -r 2 [ 1889.499105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_interrupt -r 2 [ 1889.531562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_interrupt -r 2 [ 1889.564681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_interrupt -r 2 [ 1889.629869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_interrupt -r 2 [ 1889.686653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_interrupt -r 2 [ 1889.753591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1889.807935] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1889.817932] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1890.211543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1890.238873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1890.264509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1890.361928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1890.429464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1890.485328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1890.530443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1890.584034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1890.619201] LNet: 88684:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1890.662232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1890.691443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1890.746297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1890.818607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1890.881638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1890.955609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1891.533767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_dropped -r 2 [ 1891.597541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_dropped -r 2 [ 1891.633982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_dropped -r 2 [ 1891.728421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_dropped -r 2 [ 1891.796365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_dropped -r 2 [ 1891.867935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_dropped -r 2 [ 1891.942811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_dropped -r 2 [ 1892.011890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_dropped -r 2 [ 1892.080594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1892.101763] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1892.415914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1892.462755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1892.498387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1892.542783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1892.602544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1892.652977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1892.709829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1892.751436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1892.782674] LNet: 88805:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1892.821677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1892.873987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1892.925949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1892.998705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1893.037576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1893.095820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1893.711982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_aborted -r 2 [ 1893.744646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_aborted -r 2 [ 1893.779072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_aborted -r 2 [ 1893.838485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_aborted -r 2 [ 1893.927716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_aborted -r 2 [ 1893.987690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_aborted -r 2 [ 1894.023761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_aborted -r 2 [ 1894.057553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_aborted -r 2 [ 1894.124730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1894.188393] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1894.563164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1894.617903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1894.677772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1894.728375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1894.786533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1894.841576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1894.903494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1894.970451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1894.991986] LNet: 88925:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1895.022661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1895.084556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1895.135926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1895.193734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1895.252924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1895.352969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1896.048739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_no_route -r 2 [ 1896.115835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_no_route -r 2 [ 1896.184401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_no_route -r 2 [ 1896.248834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_no_route -r 2 [ 1896.350044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_no_route -r 2 [ 1896.421978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_no_route -r 2 [ 1896.487416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_no_route -r 2 [ 1896.547189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_no_route -r 2 [ 1896.610562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1896.659865] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1897.009769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1897.060105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1897.104500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1897.155095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1897.197624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1897.240109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1897.279484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1897.320419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1897.343598] LNet: 89045:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1897.364960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1897.414245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1897.485584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1897.553610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1897.625954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1897.703659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1898.592937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_timeout -r 2 [ 1898.646928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_timeout -r 2 [ 1898.689370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_timeout -r 2 [ 1898.726778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_timeout -r 2 [ 1898.757629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_timeout -r 2 [ 1898.790825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_timeout -r 2 [ 1898.829785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_timeout -r 2 [ 1898.875024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_timeout -r 2 [ 1898.919619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1899.208495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1899.235367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1899.261934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1899.291663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1899.320589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1899.346918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1899.395561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1899.443196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1899.493994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1899.553849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1899.616686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1899.665795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1899.705483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1899.768094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1900.489313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_dropped -r 2 [ 1900.543368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e remote_dropped -r 2 [ 1900.601765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_dropped -r 2 [ 1900.651844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e remote_dropped -r 2 [ 1900.691948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e remote_dropped -r 2 [ 1900.757757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e remote_dropped -r 2 [ 1900.857558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e remote_dropped -r 2 [ 1900.916929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e remote_dropped -r 2 [ 1901.011743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1901.066737] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1901.082646] LNet: Skipped 1 previous similar message [ 1901.088529] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.154@tcp [ 1901.470901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1901.530223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1901.620040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1901.683747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1901.723593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1901.774452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1901.844642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1901.888592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1901.919815] LNet: 89285:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1901.926431] LNet: 89285:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1901.942369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1901.986435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1902.043353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1902.115955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1902.169780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1902.227953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1909.169770] LNet: 89360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.177549] LNetError: 89360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.240990] LNet: Removed LNI 192.168.204.54@tcp [ 1910.243994] LNetError: 89360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.278832] LNet: Removed LNI 192.168.204.54@tcp1 [ 1910.785862] Key type .llcrypt unregistered [ 1910.791072] Key type ._llcrypt unregistered [ 1927.368587] Key type ._llcrypt registered [ 1927.370962] Key type .llcrypt registered [ 1929.008299] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:18:35 (1788844715) [ 1929.487873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1929.514098] alg: No test for adler32 (adler32-zlib) [ 1930.571490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.709891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1930.769481] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1931.916078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1931.970881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.54@tcp [ 1932.008631] LNetError: 90338:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.54@tcp to recovery [ 1932.147735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.54@tcp [ 1932.257250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1941.165791] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:18:47 (1788844727) [ 1941.440620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1941.498075] LNet: 90924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.513667] LNetError: 90924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.571563] LNet: Removed LNI 192.168.204.54@tcp [ 1941.593704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.680347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.716029] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1942.991934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1943.037869] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1944.762655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.54@tcp [ 1944.833659] LNetError: 91194:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.54@tcp to recovery [ 1945.759367] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1945.764948] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1946.239849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.54@tcp [ 1958.785658] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:19:05 (1788844745) [ 1960.983426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg454-server [ 1962.936699] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:19:09 (1788844749) [ 1965.490770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg454-server [ 1967.473769] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:19:14 (1788844754) [ 1969.782250] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg454-server [ 1972.200210] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:19:18 (1788844758) [ 1972.355509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.380488] LNet: 92824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.407186] LNetError: 92824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.427279] LNet: Removed LNI 192.168.204.54@tcp [ 1972.447920] LNet: Removed LNI 192.168.204.54@tcp1 [ 1972.467248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.596384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1972.654200] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1973.847722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1973.969575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1974.054938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1974.258878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1974.321486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1974.469591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1983.444671] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:19:29 (1788844769) [ 1983.597816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1983.685586] LNet: 93554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.703789] LNetError: 93554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.716285] LNetError: 93554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1983.788605] LNet: Removed LNI 192.168.204.54@tcp [ 1983.821286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1983.908164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1983.962547] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1992.396800] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:19:38 (1788844778) [ 1992.703079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.734424] LNet: 94260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.743867] LNetError: 94260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.766981] LNet: Removed LNI 192.168.204.54@tcp [ 1992.790814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.864337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1992.895390] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2001.005571] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:19:47 (1788844787) [ 2002.545836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2004.630670] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:19:51 (1788844791) [ 2005.229862] LNet: 95296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.234431] LNetError: 95296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.245714] LNet: Removed LNI 192.168.204.54@tcp [ 2006.062777] Key type .llcrypt unregistered [ 2006.066898] Key type ._llcrypt unregistered [ 2014.070716] Key type ._llcrypt registered [ 2014.077180] Key type .llcrypt registered [ 2014.500345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2014.516990] alg: No test for adler32 (adler32-zlib) [ 2015.571430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2015.629195] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2023.259116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 2028.304506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.154@tcp [ 2028.814146] LNet: 95711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.836571] LNetError: 95711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.866504] LNet: Removed LNI 192.168.204.54@tcp [ 2029.704630] Key type .llcrypt unregistered [ 2029.711462] Key type ._llcrypt unregistered [ 2048.083192] Key type ._llcrypt registered [ 2048.088071] Key type .llcrypt registered [ 2050.568421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2052.963379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2055.576048] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:20:41 (1788844841) [ 2058.005817] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2061.502296] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:20:47 (1788844847) [ 2063.955086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg454-server [ 2066.506790] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:20:52 (1788844852) [ 2069.322557] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg454-server [ 2071.636619] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:20:58 (1788844858) [ 2073.507215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2073.546451] alg: No test for adler32 (adler32-zlib) [ 2074.623066] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 2074.841063] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2076.504524] Key type lgssc registered [ 2077.923624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2100.399706] Key type lgssc unregistered [ 2100.744803] LNet: 98885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.759217] LNetError: 98885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.789044] LNet: Removed LNI 192.168.204.54@tcp [ 2101.691236] Key type .llcrypt unregistered [ 2101.694968] Key type ._llcrypt unregistered [ 2117.837524] Key type ._llcrypt registered [ 2117.841058] Key type .llcrypt registered [ 2119.175589] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:21:45 (1788844905) [ 2119.753901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.781969] alg: No test for adler32 (adler32-zlib) [ 2120.864905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.960520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2121.018596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2121.067579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2121.143390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2128.857392] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:21:55 (1788844915) [ 2129.206892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.228949] LNet: 100297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.246786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.308592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2129.329223] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2130.499179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2130.523731] LNet: 100442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2130.531159] LNetError: 100442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2130.578984] LNet: Removed LNI 192.168.204.54@tcp [ 2130.603359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.691176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2130.710921] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2131.344965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.387111] LNetError: 100470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.423422] LNet: Removed LNI 192.168.204.54@tcp [ 2131.453599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.525594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2131.549807] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2138.951782] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:22:05 (1788844925) [ 2139.360597] LNet: 101060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.372279] LNet: 101060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2139.378254] LNetError: 101060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.395380] LNet: Removed LNI 192.168.204.54@tcp [ 2140.198608] Key type .llcrypt unregistered [ 2140.203529] Key type ._llcrypt unregistered [ 2147.783635] Key type ._llcrypt registered [ 2147.785825] Key type .llcrypt registered [ 2148.085468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2148.134627] alg: No test for adler32 (adler32-zlib) [ 2149.149858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.308929] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2149.316548] LNet: *** cfs_fail_loc=e003, val=100*** [ 2150.135610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2150.157614] LNet: *** cfs_fail_loc=e003, val=100*** [ 2150.161099] LNet: Skipped 2 previous similar messages [ 2150.175048] Ping reply received with mapping: [ 2150.177067] found metadata reply NI[192.168.204.54@tcp] [ 2150.180194] 36 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2150.186682] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2150.192207] 5a 5a 5a 5a 5a 5a 5a 5a [ 2150.386771] LNet: 101454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.408915] LNetError: 101454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.429834] LNet: Removed LNI 192.168.204.54@tcp [ 2151.228752] Key type .llcrypt unregistered [ 2151.233722] Key type ._llcrypt unregistered [ 2164.233400] Key type ._llcrypt registered [ 2164.237301] Key type .llcrypt registered [ 2165.978903] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:22:32 (1788844952) [ 2166.697286] Key type .llcrypt unregistered [ 2166.699408] Key type ._llcrypt unregistered [ 2174.471514] Key type ._llcrypt registered [ 2174.481699] Key type .llcrypt registered [ 2174.795718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2174.819947] alg: No test for adler32 (adler32-zlib) [ 2176.276417] LNetError: 102492:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2176.762730] Key type .llcrypt unregistered [ 2176.765254] Key type ._llcrypt unregistered [ 2183.280321] Key type ._llcrypt registered [ 2183.283801] Key type .llcrypt registered [ 2184.684294] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:22:51 (1788844971) [ 2185.422489] Key type .llcrypt unregistered [ 2185.424811] Key type ._llcrypt unregistered [ 2193.946060] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2193.970878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2196.679295] Key type ._llcrypt registered [ 2196.681129] Key type .llcrypt registered [ 2196.921552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2196.933574] alg: No test for adler32 (adler32-zlib) [ 2197.858940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.975331] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2208.010874] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:23:14 (1788844994) [ 2208.241787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.259990] LNet: 104268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.265886] LNetError: 104268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2208.280629] LNet: Removed LNI 10.1.2.3@tcp [ 2208.303237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.040665] Key type .llcrypt unregistered [ 2209.046134] Key type ._llcrypt unregistered [ 2216.880288] Key type ._llcrypt registered [ 2216.884195] Key type .llcrypt registered [ 2217.036511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2217.046709] alg: No test for adler32 (adler32-zlib) [ 2218.084494] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2218.152713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2218.353916] LNet: 104554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.359605] LNetError: 104554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.372905] LNet: Removed LNI 192.168.204.54@tcp [ 2219.328488] Key type .llcrypt unregistered [ 2219.332185] Key type ._llcrypt unregistered [ 2233.589791] Key type ._llcrypt registered [ 2233.594541] Key type .llcrypt registered [ 2235.635209] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:23:41 (1788845021) [ 2236.461697] Key type .llcrypt unregistered [ 2236.465745] Key type ._llcrypt unregistered [ 2245.106589] Key type ._llcrypt registered [ 2245.111215] Key type .llcrypt registered [ 2245.558128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2245.603267] alg: No test for adler32 (adler32-zlib) [ 2254.171833] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:24:00 (1788845040) [ 2254.276822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2254.318821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.428685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2254.464625] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2255.265306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2255.313279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2255.364397] LNet: Removed LNI 192.168.204.54@tcp [ 2255.700158] LNet: 106296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.310078] Key type .llcrypt unregistered [ 2256.318311] Key type ._llcrypt unregistered [ 2270.701299] Key type ._llcrypt registered [ 2270.706328] Key type .llcrypt registered [ 2272.930315] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:24:18 (1788845058) [ 2273.475933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2273.507741] alg: No test for adler32 (adler32-zlib) [ 2274.554377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.651652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2274.697691] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2276.032055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2276.108570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.54@tcp [ 2276.156488] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2276.168110] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2276.178662] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2276, 0, 0 [ 2276.188208] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2276.193445] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2276.201515] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2276.209788] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2276, 0, 0 [ 2276.222874] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2276.230542] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2276.234758] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2276.240410] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2276, 0, 2326 [ 2276.247788] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2276.252270] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2276.256718] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2276.264534] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2276, 0, 2326 [ 2276.275320] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2285.353531] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:24:31 (1788845071) [ 2285.885376] LNet: 107835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.897043] LNetError: 107835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.927213] LNet: Removed LNI 192.168.204.54@tcp [ 2286.792585] Key type .llcrypt unregistered [ 2286.795161] Key type ._llcrypt unregistered [ 2294.220562] Key type ._llcrypt registered [ 2294.222033] Key type .llcrypt registered [ 2294.540977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2294.562368] alg: No test for adler32 (adler32-zlib) [ 2295.657362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2295.708166] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2305.119959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 2316.385859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.444276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2316.490223] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 2317.911968] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2318.035676] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2318.039329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2318.046176] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2320.781909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2320.838326] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2335.253142] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:25:21 (1788845121) [ 2335.444564] LNet: 109343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.484608] LNetError: 109343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.543180] LNet: Removed LNI 192.168.204.54@tcp [ 2336.553202] LNetError: 109343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.567067] LNet: Removed LNI 192.168.204.54@tcp1 [ 2336.586028] LNet: Removed LNI 10.1.2.3@tcp99 [ 2337.416622] Key type .llcrypt unregistered [ 2337.420415] Key type ._llcrypt unregistered [ 2345.969795] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2345.978309] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2348.787750] Key type ._llcrypt registered [ 2348.789863] Key type .llcrypt registered [ 2349.096286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2349.127509] alg: No test for adler32 (adler32-zlib) [ 2350.063325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.128321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2350.163270] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2351.334252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2351.373260] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2354.758368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2354.848137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2354.914463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2354.939954] LNetError: 110093:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2354.992676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2355.050171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2355.876525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2356.003873] LNet: 110113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.015762] LNetError: 110113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2356.030791] LNet: Removed LNI 10.1.2.3@tcp [ 2357.103083] LNet: Removed LNI 192.168.204.54@tcp [ 2357.986118] Key type .llcrypt unregistered [ 2357.995099] Key type ._llcrypt unregistered [ 2372.158411] Key type ._llcrypt registered [ 2372.162517] Key type .llcrypt registered [ 2373.505949] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:26:00 (1788845160) [ 2374.035170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.064744] alg: No test for adler32 (adler32-zlib) [ 2375.108080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.216843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2375.264466] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2376.204711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2376.323808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg454-client@tcp [ 2384.593545] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:26:11 (1788845171) [ 2384.923772] LNet: 111644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2384.935802] LNetError: 111644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2384.949427] LNet: Removed LNI 192.168.204.54@tcp [ 2385.749514] Key type .llcrypt unregistered [ 2385.753528] Key type ._llcrypt unregistered [ 2393.564852] Key type ._llcrypt registered [ 2393.568226] Key type .llcrypt registered [ 2393.815188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2393.835489] alg: No test for adler32 (adler32-zlib) [ 2394.776503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2394.799671] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2394.828745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2402.012717] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:26:28 (1788845188) [ 2403.720111] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2405.810390] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:26:32 (1788845192) [ 2407.473488] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2409.772598] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:26:36 (1788845196) [ 2410.102810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.128798] LNet: 113064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2410.135724] LNetError: 113064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2410.152259] LNet: Removed LNI 192.168.204.54@tcp [ 2410.170235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.272118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2410.315767] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2411.277162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2411.358693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2418.762860] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:26:45 (1788845205) [ 2418.971555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2418.989499] LNet: 113774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2418.998250] LNetError: 113774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.073839] LNet: Removed LNI 192.168.204.54@tcp [ 2420.137760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2420.218535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2420.292400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2420.594202] LNet: 113810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.154439] Key type .llcrypt unregistered [ 2421.156497] Key type ._llcrypt unregistered [ 2434.135216] Key type ._llcrypt registered [ 2434.137480] Key type .llcrypt registered [ 2435.730657] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:27:02 (1788845222) [ 2436.156102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2436.176281] alg: No test for adler32 (adler32-zlib) [ 2437.126354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.223782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2437.293561] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2438.175654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2438.245462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2445.745649] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:27:12 (1788845232) [ 2446.074965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2446.101571] LNet: 115328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.111092] LNetError: 115328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2446.124498] LNet: Removed LNI 192.168.204.54@tcp [ 2446.166512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.221334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2446.253407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2453.433448] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:27:19 (1788845239) [ 2453.549450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2453.575972] LNet: 115903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2453.646557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2453.755539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2454.273810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2454.321480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2454.394413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2454.804889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2454.840546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2454.911531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2454.925533] LNet: 115941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.941737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2454.995758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2455.458989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2455.516845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2455.591261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2456.078298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2456.121400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2462.766943] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:27:29 (1788845249) [ 2463.125462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2463.155552] LNet: 116542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.184086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2463.237912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2463.264698] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2463.954300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2463.977478] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2464.920466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2464.988333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2465.304973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2465.336502] LNetError: 116821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2465.350868] LNet: Removed LNI 192.168.204.54@tcp [ 2465.372880] LNet: Removed LNI 192.168.204.54@tcp2 [ 2465.409533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2465.493130] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2465.501821] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2465.518083] LNet: 116827:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2465.991811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2466.271640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2467.359591] LNet: Removed LNI 192.168.204.54@tcp [ 2467.361805] LNetError: 116864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.375053] LNetError: 116864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2467.394558] LNet: Removed LNI 192.168.204.54@tcp2 [ 2467.449535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2467.517569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2467.552519] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2468.551676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2468.578258] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2469.344217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2469.396222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2469.498952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2469.752316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2469.781405] LNet: 117149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.787271] LNet: 117149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2469.798171] LNetError: 117149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.817219] LNet: Removed LNI 192.168.204.54@tcp [ 2470.886907] LNet: Removed LNI 192.168.204.54@tcp2 [ 2470.906694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2471.017118] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2471.035348] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2471.043304] LNet: 117155:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2471.572425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2471.892285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2471.917880] LNet: Removed LNI 192.168.204.54@tcp [ 2472.936463] LNet: Removed LNI 192.168.204.54@tcp2 [ 2472.993891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2473.092774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2473.148308] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2473.999280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2474.016346] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2475.077165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2475.175931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2475.383625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2475.402638] LNetError: 117471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2475.408240] LNetError: 117471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2475.414189] LNet: Removed LNI 192.168.204.54@tcp [ 2475.428776] LNet: Removed LNI 192.168.204.54@tcp2 [ 2475.449534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.505983] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2475.510462] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2475.514728] LNet: 117478:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2476.014187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2476.185369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2477.215148] LNet: Removed LNI 192.168.204.54@tcp [ 2477.239458] LNet: Removed LNI 192.168.204.54@tcp2 [ 2477.281866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2477.348300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2477.369354] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2478.067492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2478.099752] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2478.947821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2478.995405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2479.064713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2479.265446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2479.301031] LNet: 117799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2479.305811] LNet: 117799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2480.351261] LNet: Removed LNI 192.168.204.54@tcp [ 2481.381795] LNet: Removed LNI 192.168.204.54@tcp2 [ 2481.439631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2481.535650] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2481.548822] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2481.555548] LNet: 117805:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2482.100728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2482.305765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2482.320739] LNet: Removed LNI 192.168.204.54@tcp [ 2483.367941] LNet: Removed LNI 192.168.204.54@tcp2 [ 2483.441210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2483.514871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2483.537510] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2484.302711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2484.325463] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2485.133292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2485.183588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2485.234985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2485.569809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2485.600839] LNetError: 118140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2485.613282] LNetError: 118140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2485.624492] LNet: Removed LNI 192.168.204.54@tcp [ 2485.651110] LNet: Removed LNI 192.168.204.54@tcp2 [ 2485.680346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2485.740057] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2485.747125] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2485.753925] LNet: 118146:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2486.503577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2487.019474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2488.095266] LNet: Removed LNI 192.168.204.54@tcp [ 2488.111451] LNet: Removed LNI 192.168.204.54@tcp2 [ 2488.189799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2488.347421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2488.381424] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2489.061748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2489.098437] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2490.103360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2490.164392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2490.220850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2490.299503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2490.714611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2491.807218] LNet: Removed LNI 192.168.204.54@tcp [ 2492.840789] LNet: Removed LNI 192.168.204.54@tcp2 [ 2492.859462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.907692] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2492.912562] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2493.561153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2501.232061] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2503.009969] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:28:09 (1788845289) [ 2503.148255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2503.178178] LNet: 119259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2503.185333] LNet: 119259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2503.195794] LNetError: 119259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2503.204703] LNetError: 119259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2503.215817] LNet: Removed LNI 192.168.204.54@tcp [ 2504.231031] LNet: Removed LNI 192.168.204.54@tcp2 [ 2504.299989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2504.579741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2504.615571] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2505.295529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2505.312074] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2506.134336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2506.157575] LNet: Removed LNI 10.1.2.3@tcp [ 2506.165633] LNet: Removed LNI 192.168.204.54@tcp [ 2506.194197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2506.276559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2506.326759] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2506.331711] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2506.789335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2506.812325] LNet: Removed LNI 192.168.204.54@tcp [ 2506.831924] LNet: Removed LNI 10.1.2.3@tcp [ 2514.094036] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:28:20 (1788845300) [ 2514.434508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2514.480397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2514.580338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2514.608287] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2515.730437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2515.859548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2515.910645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2515.959082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2515.996438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2516.046822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2516.092774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2516.139721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2516.191573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2516.263922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2516.330503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2516.580447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2524.927326] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:28:31 (1788845311) [ 2525.036511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2526.134443] LNet: Removed LNI 192.168.204.54@tcp [ 2526.180607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2526.346503] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2526.477727] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2526.485301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2526.489288] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2529.118918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2529.165627] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2530.279504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2530.308081] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2533.418388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2533.502738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2533.606992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.54@tcp -m PUT -e local_timeout -r 1 [ 2533.710987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2533.809661] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2533.815160] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2534.070152] LNetError: 120944:0:(lib-move.c:4801:lnet_parse()) 192.168.204.54@tcp, src 192.168.204.54@tcp: Dropping PUT (error -108 looking up sender) [ 2535.903223] LNet: Removed LNI 10.1.2.3@tcp [ 2535.907140] LNetError: 121403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.916207] LNetError: 121403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2535.956324] LNet: Removed LNI 192.168.204.54@tcp [ 2536.502112] Key type .llcrypt unregistered [ 2536.505849] Key type ._llcrypt unregistered [ 2550.051892] Key type ._llcrypt registered [ 2550.057340] Key type .llcrypt registered [ 2551.648598] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:28:58 (1788845338) [ 2552.227807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2552.252433] alg: No test for adler32 (adler32-zlib) [ 2553.289699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2553.818633] LNet: 122254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2554.636438] Key type .llcrypt unregistered [ 2554.638175] Key type ._llcrypt unregistered [ 2563.607910] Key type ._llcrypt registered [ 2563.609668] Key type .llcrypt registered [ 2563.991923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2564.033593] alg: No test for adler32 (adler32-zlib) [ 2565.081144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2565.135863] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2573.849071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 2574.073728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2575.033864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.154@tcp [ 2575.340977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2575.368855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2575.392135] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.154@tcp [ 2575.404661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2575.445913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2575.492950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2575.536816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2575.577779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2575.625357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2575.648239] LNet: 122700:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2575.667323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2575.699232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2575.730230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2575.771829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2575.844343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2575.960533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2577.051333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2577.464956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2577.525314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2577.570465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2577.620197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2577.661465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2577.704281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2577.759888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2577.883602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2577.928941] LNet: 122817:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2577.973377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2578.051242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2578.173723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2578.234505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2578.284587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2578.344812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2579.495916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.154@tcp [ 2579.814449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2579.876313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2579.917139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2579.957833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2580.005095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2580.058808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2580.124354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2580.194775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2580.227809] LNet: 122935:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2580.248746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2580.282737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2580.314671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2580.348832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2580.390465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2580.448470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2581.424638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2581.825356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2581.882868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2581.917323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2581.953807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2582.012583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2582.062501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2582.119864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2582.165103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2582.237964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2582.316624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2582.375664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2582.423237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2582.477857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2582.541603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2582.918813] LNet: 123098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2582.928842] LNetError: 123098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2582.955057] LNet: Removed LNI 192.168.204.54@tcp [ 2583.340975] Key type .llcrypt unregistered [ 2583.354375] Key type ._llcrypt unregistered [ 2604.304185] Key type ._llcrypt registered [ 2604.306532] Key type .llcrypt registered [ 2606.343994] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:29:52 (1788845392) [ 2606.775859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2606.792957] alg: No test for adler32 (adler32-zlib) [ 2607.780898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2608.386612] LNet: 123983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2609.235414] Key type .llcrypt unregistered [ 2609.237133] Key type ._llcrypt unregistered [ 2616.103614] Key type ._llcrypt registered [ 2616.107352] Key type .llcrypt registered [ 2616.473175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2616.520327] alg: No test for adler32 (adler32-zlib) [ 2617.509459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2617.556708] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2625.315659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 2634.865729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2634.919601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2634.952622] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 2635.961578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2636.545586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.154@tcp [ 2636.803191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2636.845537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2636.890492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2636.958963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2637.012793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2637.024124] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.154@tcp [ 2637.057330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2637.097405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2637.148595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2637.200277] LNet: 124633:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2637.237091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2637.295826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2637.349851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2637.395600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2637.441876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2637.501485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2638.321295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2638.651970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2638.691375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2638.736260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2638.771499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2638.805269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2638.857621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2638.898199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2638.943847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2638.971097] LNet: 124750:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2638.997049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2639.034561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2639.078271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2639.119363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2639.157867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2639.204372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2639.974962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.154@tcp [ 2640.264706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2640.306673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2640.345417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2640.394736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2640.448434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2640.502865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2640.538248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2640.573734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2640.589281] LNet: 124867:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2640.610565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2640.671927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2640.720925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2640.777696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2640.805277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2640.845102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2641.649679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2641.989626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2642.033180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2642.078939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2642.107421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2642.157774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2642.201938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2642.281877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2642.343584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2642.412687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2642.475976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2642.536648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2642.600047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2642.672956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2642.725411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2648.719979] LNet: 125065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2648.734408] LNetError: 125065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2648.739668] LNet: Removed LNI 192.168.204.54@tcp [ 2648.755340] LNet: Removed LNI 192.168.204.54@tcp1 [ 2649.175642] Key type .llcrypt unregistered [ 2649.187919] Key type ._llcrypt unregistered [ 2669.289302] Key type ._llcrypt registered [ 2669.295873] Key type .llcrypt registered [ 2671.000913] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 01:30:57 (1788845457) [ 2671.594969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2671.620423] alg: No test for adler32 (adler32-zlib) [ 2672.609786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2672.691830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2672.737696] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2674.054973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2674.095276] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2675.394789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2675.519772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2676.575644] LNet: Removed LNI 192.168.204.54@tcp [ 2683.998604] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:31:10 (1788845470) [ 2685.788598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2688.052764] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:31:14 (1788845474) [ 2696.839273] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:31:23 (1788845483) [ 2706.109816] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:31:32 (1788845492) [ 2713.636245] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:31:40 (1788845500) [ 2722.342539] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:31:48 (1788845508) [ 2729.709753] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:31:56 (1788845516) [ 2737.322654] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:32:03 (1788845523) [ 2744.431613] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:32:11 (1788845531) [ 2752.523785] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:32:19 (1788845539) [ 2759.375085] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:32:26 (1788845546) [ 2765.666676] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:32:32 (1788845552) [ 2773.349504] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:32:39 (1788845559) [ 2779.616133] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:32:46 (1788845566) [ 2786.786605] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:32:53 (1788845573) [ 2794.643793] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:33:01 (1788845581) [ 2802.003489] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:33:08 (1788845588) [ 2809.896556] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:33:16 (1788845596) [ 2817.622734] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:33:24 (1788845604) [ 2817.746273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2817.787625] LNet: 136750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2817.805545] LNetError: 136750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2817.832625] LNet: Removed LNI 192.168.204.54@tcp2 [ 2817.870684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2818.046577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2818.067480] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2819.817570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2819.865069] LNet: 136899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2819.882965] LNetError: 136899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2819.895868] LNet: Removed LNI 192.168.204.54@tcp [ 2819.918993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2820.009403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2820.051031] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2827.657272] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:33:34 (1788845614) [ 2835.786623] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:33:42 (1788845622) [ 2844.230175] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:33:50 (1788845630) [ 2854.306492] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:34:00 (1788845640) [ 2861.373938] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:34:08 (1788845648) [ 2869.235498] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:34:15 (1788845655) [ 2877.044898] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:34:23 (1788845663) [ 2884.348538] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:34:31 (1788845671) [ 2891.778510] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:34:38 (1788845678) [ 2898.786601] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:34:45 (1788845685) [ 2906.426661] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:34:53 (1788845693) [ 2913.109380] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:35:00 (1788845700) [ 2921.250394] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2665 sec ========== 01:35:08 (1788845708) [ 2921.615598] LNet: 144364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2921.624647] LNetError: 144364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2921.643919] LNet: Removed LNI 192.168.204.54@tcp [ 2922.294454] Key type .llcrypt unregistered [ 2922.299483] Key type ._llcrypt unregistered [ 2934.957675] Key type ._llcrypt registered [ 2934.962348] Key type .llcrypt registered [ 2935.748812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2935.764673] alg: No test for adler32 (adler32-zlib) [ 2936.935174] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 2937.180758] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2938.848446] Key type lgssc registered [ 2941.093891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3013.965609] Lustre: Mounted lustre-client - version 2.17.57_112_g92cfe64 [ 3019.813984] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3030.796622] Lustre: Unmounted lustre-client [ 3062.817664] Key type lgssc unregistered [ 3063.193377] LNet: 147316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3063.204378] LNetError: 147316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3063.241713] LNet: Removed LNI 192.168.204.54@tcp [ 3064.385454] Key type .llcrypt unregistered [ 3064.388610] Key type ._llcrypt unregistered