[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 427371973 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003163] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009019] pid_max: default: 32768 minimum: 301 [ 0.010153] LSM: Security Framework initializing [ 0.011068] Yama: becoming mindful. [ 0.012000] SELinux: Initializing. [ 0.012090] *** VALIDATE selinux *** [ 0.021707] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026540] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027200] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029126] *** VALIDATE tmpfs *** [ 0.031165] *** VALIDATE proc *** [ 0.032293] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034308] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036019] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038035] Spectre V2 : User space: Vulnerable [ 0.039013] Speculative Store Bypass: Vulnerable [ 0.042709] debug: unmapping init [mem 0xffffffff84c59000-0xffffffff84c60fff] [ 0.044187] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045748] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046034] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048017] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053354] rcu: Hierarchical SRCU implementation. [ 0.055621] smp: Bringing up secondary CPUs ... [ 0.056596] x86: Booting SMP configuration: [ 0.057031] .... node #0, CPUs: #1 #2 #3 [ 0.061088] smp: Brought up 1 node, 4 CPUs [ 0.063021] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129958] node 0 deferred pages initialised in 63ms [ 0.133011] devtmpfs: initialized [ 0.134211] x86/mm: Memory block size: 128MB [ 0.136423] gcov: version magic: 0x41383552 [ 0.138179] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142134] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144283] pinctrl core: initialized pinctrl subsystem [ 0.146160] [ 0.146648] ************************************************************* [ 0.148011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149009] ** ** [ 0.151009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152052] ** ** [ 0.153011] ** This means that this kernel is built to expose internal ** [ 0.155012] ** IOMMU data structures, which may compromise security on ** [ 0.156056] ** your system. ** [ 0.158023] ** ** [ 0.159012] ** If you see this message and you are not debugging the ** [ 0.161011] ** kernel, report this immediately to your vendor! ** [ 0.162011] ** ** [ 0.164009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166011] ************************************************************* [ 0.167855] NET: Registered protocol family 16 [ 0.170395] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175351] cpuidle: using governor menu [ 0.176511] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177403] PCI: Using configuration type 1 for base access [ 0.178000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188122] cryptd: max_cpu_qlen set to 1000 [ 0.190256] ACPI: Added _OSI(Module Device) [ 0.191015] ACPI: Added _OSI(Processor Device) [ 0.192011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197311] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.202526] ACPI: Interpreter enabled [ 0.204059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205013] ACPI: Using IOAPIC for interrupt routing [ 0.207109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209390] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217474] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.224099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229315] acpiphp: Slot [2] registered [ 0.230077] acpiphp: Slot [5] registered [ 0.231121] acpiphp: Slot [6] registered [ 0.232160] acpiphp: Slot [3] registered [ 0.232952] acpiphp: Slot [4] registered [ 0.234106] acpiphp: Slot [7] registered [ 0.235127] acpiphp: Slot [8] registered [ 0.236103] acpiphp: Slot [9] registered [ 0.238134] acpiphp: Slot [10] registered [ 0.240094] acpiphp: Slot [11] registered [ 0.241092] acpiphp: Slot [12] registered [ 0.242014] acpiphp: Slot [13] registered [ 0.242970] acpiphp: Slot [14] registered [ 0.243079] acpiphp: Slot [15] registered [ 0.244041] acpiphp: Slot [16] registered [ 0.247122] acpiphp: Slot [17] registered [ 0.247937] acpiphp: Slot [18] registered [ 0.249108] acpiphp: Slot [19] registered [ 0.250008] acpiphp: Slot [20] registered [ 0.250996] acpiphp: Slot [21] registered [ 0.253108] acpiphp: Slot [22] registered [ 0.255147] acpiphp: Slot [23] registered [ 0.256103] acpiphp: Slot [24] registered [ 0.258093] acpiphp: Slot [25] registered [ 0.260094] acpiphp: Slot [26] registered [ 0.261154] acpiphp: Slot [27] registered [ 0.264178] acpiphp: Slot [28] registered [ 0.266149] acpiphp: Slot [29] registered [ 0.267111] acpiphp: Slot [30] registered [ 0.269132] acpiphp: Slot [31] registered [ 0.270101] PCI host bridge to bus 0000:00 [ 0.272035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.274025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.277025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.279026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.282029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.287218] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.291242] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.295362] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.303020] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.307057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.310024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.312020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.314021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.315495] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.317809] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.320042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.322584] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.326019] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.337019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.341014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.347263] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.353015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.357012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.365017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.372957] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.377013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.384016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.397023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.405937] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.408403] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.410265] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.412290] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.413138] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.418144] iommu: Default domain type: Passthrough [ 0.421565] SCSI subsystem initialized [ 0.422139] ACPI: bus type USB registered [ 0.424094] usbcore: registered new interface driver usbfs [ 0.425110] usbcore: registered new interface driver hub [ 0.426077] usbcore: registered new device driver usb [ 0.428178] pps_core: LinuxPPS API ver. 1 registered [ 0.429008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.431071] PTP clock support registered [ 0.432183] EDAC MC: Ver: 3.0.0 [ 0.433100] PCI: Using ACPI for IRQ routing [ 0.434524] NetLabel: Initializing [ 0.435008] NetLabel: domain hash size = 128 [ 0.435850] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.437068] NetLabel: unlabeled traffic allowed by default [ 0.439134] vgaarb: loaded [ 0.440194] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.441009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.447360] clocksource: Switched to clocksource kvm-clock [ 0.553697] VFS: Disk quotas dquot_6.6.0 [ 0.555743] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.558251] *** VALIDATE ramfs *** [ 0.559632] *** VALIDATE hugetlbfs *** [ 0.561285] pnp: PnP ACPI init [ 0.563612] pnp: PnP ACPI: found 6 devices [ 0.579379] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.582907] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.585204] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.586938] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.589474] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.592048] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.594954] NET: Registered protocol family 2 [ 0.597427] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.602601] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.606348] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.611360] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.614749] TCP: Hash tables configured (established 65536 bind 65536) [ 0.617385] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.619696] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.622242] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.624877] NET: Registered protocol family 1 [ 0.627228] RPC: Registered named UNIX socket transport module. [ 0.629306] RPC: Registered udp transport module. [ 0.630740] RPC: Registered tcp transport module. [ 0.632215] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.634285] NET: Registered protocol family 44 [ 0.635855] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.638030] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.639948] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.641956] PCI: CLS 0 bytes, default 64 [ 0.643464] Unpacking initramfs... [ 1.977730] debug: unmapping init [mem 0xffff9d977cc64000-0xffff9d977ffcffff] [ 1.981707] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.984058] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.986781] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.499136] Initialise system trusted keyrings [ 2.500985] Key type blacklist registered [ 2.503631] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.513096] zbud: loaded [ 2.516321] *** VALIDATE nfs *** [ 2.517605] *** VALIDATE nfs4 *** [ 2.519336] pstore: using deflate compression [ 2.523644] Platform Keyring initialized [ 2.635728] NET: Registered protocol family 38 [ 2.636965] Key type asymmetric registered [ 2.637887] Asymmetric key parser 'x509' registered [ 2.638993] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.640927] io scheduler mq-deadline registered [ 2.643012] io scheduler kyber registered [ 2.645104] io scheduler bfq registered [ 2.647920] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.650070] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.651790] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.653973] ACPI: Power Button [PWRF] [ 2.659557] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.667569] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.680198] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.708621] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.737288] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.741646] Non-volatile memory driver v1.3 [ 2.742657] Linux agpgart interface v0.103 [ 2.768211] virtio_blk virtio1: [vda] 150040 512-byte logical blocks (76.8 MB/73.3 MiB) [ 2.771046] vda: detected capacity change from 0 to 76820480 [ 2.787278] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.789035] vdb: detected capacity change from 0 to 1073741824 [ 2.797385] libphy: Fixed MDIO Bus: probed [ 2.805498] usbcore: registered new interface driver usbserial_generic [ 2.806993] usbserial: USB Serial support registered for generic [ 2.808372] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.811325] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.812394] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.814055] mousedev: PS/2 mouse device common for all mice [ 2.816160] rtc_cmos 00:05: RTC can wake from S4 [ 2.817516] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.820138] rtc_cmos 00:05: registered as rtc0 [ 2.822750] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.825580] intel_pstate: CPU model not supported [ 2.828884] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.833643] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.833671] hid: raw HID events driver (C) Jiri Kosina [ 2.838419] usbcore: registered new interface driver usbhid [ 2.840715] usbhid: USB HID core driver [ 2.842596] drop_monitor: Initializing network drop monitor service [ 2.845159] Initializing XFRM netlink socket [ 2.847458] NET: Registered protocol family 10 [ 2.850694] Segment Routing with IPv6 [ 2.852402] NET: Registered protocol family 17 [ 2.854766] mpls_gso: MPLS GSO support [ 2.860224] RAS: Correctable Errors collector initialized. [ 2.862596] AVX version of gcm_enc/dec engaged. [ 2.864357] AES CTR mode by8 optimization enabled [ 2.924273] sched_clock: Marking stable (2924254679, 0)->(3843046663, -918791984) [ 2.926504] registered taskstats version 1 [ 2.927930] Loading compiled-in X.509 certificates [ 2.929965] zswap: loaded using pool lzo/zbud [ 2.951317] Key type big_key registered [ 2.963512] Key type encrypted registered [ 2.965133] ima: No TPM chip found, activating TPM-bypass! [ 2.967653] ima: Allocated hash algorithm: sha1 [ 2.969602] ima: No architecture policies found [ 2.971296] evm: Initialising EVM extended attributes: [ 2.973219] evm: security.selinux [ 2.974507] evm: security.ima [ 2.975931] evm: security.capability [ 2.977320] evm: HMAC attrs: 0x1 [ 2.980548] rtc_cmos 00:05: setting system clock to 2026-09-07 01:57:31 UTC (1788746251) [ 2.987305] debug: unmapping init [mem 0xffffffff85c03000-0xffffffff85dfffff] [ 2.990487] debug: unmapping init [mem 0xffffffff84982000-0xffffffff84c58fff] [ 3.000135] Write protecting the kernel read-only data: 28672k [ 3.002636] debug: unmapping init [mem 0xffffffff83003000-0xffffffff831fffff] [ 3.004721] debug: unmapping init [mem 0xffffffff83914000-0xffffffff839fffff] [ 3.035394] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.045352] systemd[1]: Detected virtualization kvm. [ 3.046639] systemd[1]: Detected architecture x86-64. [ 3.048052] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.075551] systemd[1]: No hostname configured. [ 3.077037] systemd[1]: Set hostname to . [ 3.079025] random: systemd: uninitialized urandom read (16 bytes read) [ 3.081289] systemd[1]: Initializing machine ID from random generator. [ 3.134227] random: ln: uninitialized urandom read (6 bytes read) [ 3.226414] random: systemd: uninitialized urandom read (16 bytes read) [ 3.229854] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.236246] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.241103] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.917144] device-mapper: uevent: version 1.0.3 [ 3.918874] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.657816] virtio_net virtio0 ens2: renamed from eth0 [ 4.716276] scsi host0: ata_piix [ 4.737877] scsi host1: ata_piix [ 4.815402] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.817931] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.565485] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.483356] random: crng init done [ 9.484733] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.023923] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.241157] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.484418] SELinux: Disabled at runtime. [ 11.544508] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.554557] systemd[1]: Detected virtualization kvm. [ 11.556899] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.996980] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.999921] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.005458] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.009294] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.013264] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.020968] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.027988] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED[ 12.054897] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.496818] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.799602] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.874721] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.969399] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.986610] EDAC sbridge: Ver: 1.1.2 [ 14.085576] Key type dns_resolver registered [ 14.380824] NFS: Registering the id_resolver key type [ 14.382483] Key type id_resolver registered [ 14.384071] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg115-client login: [ 44.169305] libcfs: loading out-of-tree module taints kernel. [ 44.231682] Key type ._llcrypt registered [ 44.233349] Key type .llcrypt registered [ 44.512650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 44.520431] alg: No test for adler32 (adler32-zlib) [ 46.031722] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 46.881416] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 48.568811] Key type lgssc registered [ 50.027983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 208.795381] Lustre: Mounted lustre-client - version 2.17.58_39_g34d8757 [ 216.120969] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.548063] hrtimer: interrupt took 9068994 ns [ 230.739982] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing check_logdir /tmp/testlogs/ [ 234.463516] Lustre: lustre-OST0000-osc-ffff9d97d2bf7800: disconnect after 22s idle [ 236.826191] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing yml_node [ 242.556723] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 245.295129] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 248.812315] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 251.356638] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Sep 6 22:01:37 EDT 2026 [ 270.309221] Lustre: Unmounted lustre-client [ 289.075319] Key type lgssc unregistered [ 289.451894] LNet: 8423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.466204] LNetError: 8423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.493379] LNet: Removed LNI 192.168.201.15@tcp [ 290.509363] Key type .llcrypt unregistered [ 290.511898] Key type ._llcrypt unregistered [ 316.099506] Key type ._llcrypt registered [ 316.108991] Key type .llcrypt registered [ 316.378377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 316.412258] alg: No test for adler32 (adler32-zlib) [ 317.435399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 317.529712] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 317.570990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 322.251671] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 322.360946] LNet: 9294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.369512] LNetError: 9294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 322.397392] LNet: Removed LNI 192.168.201.15@tcp [ 323.197685] Key type .llcrypt unregistered [ 323.200077] Key type ._llcrypt unregistered [ 329.502705] Key type ._llcrypt registered [ 329.507724] Key type .llcrypt registered [ 330.825945] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 22:02:57 (1788746577) [ 331.340697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 331.354591] alg: No test for adler32 (adler32-zlib) [ 332.279683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.362507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.379512] LNet: 9711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.397607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.442317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 332.459781] LNet: 9718:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 338.460434] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 22:03:05 (1788746585) [ 338.548266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.570329] LNet: 10291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.596541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.702427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 345.274420] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 22:03:12 (1788746592) [ 345.707480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.779606] LNet: 10884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.798244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.896712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 351.872640] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 22:03:19 (1788746599) [ 352.010110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.034039] LNet: 11468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.085249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.180880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 358.931126] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 22:03:26 (1788746606) [ 359.002487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.032244] LNet: 12047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.053948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.164526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 359.628399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 365.799149] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 22:03:33 (1788746613) [ 365.903402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.988919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.069894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 366.558932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 367.083718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 374.384564] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 22:03:41 (1788746621) [ 374.468049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.496181] LNet: 13234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.501763] LNet: 13234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 374.532285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.644676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 382.145628] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 22:03:48 (1788746628) [ 382.274880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.360695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.482738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 382.515714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 383.010816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 383.042688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 383.756614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 383.788889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 384.217563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 384.274326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 384.884821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 384.916789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 385.415864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 385.490929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 386.144602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 386.176321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 386.594452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 386.643481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 387.270253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 387.326533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 387.777588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 387.800486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 388.169585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 388.212769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 388.643477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 388.679541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 388.998472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 389.039595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 395.909503] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 22:04:03 (1788746643) [ 395.995954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.020810] LNet: 14500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.031876] LNet: 14500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 396.065949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.170433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 396.208977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 402.529339] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 22:04:09 (1788746649) [ 402.768371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.809217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.881966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 402.910482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 409.779983] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 22:04:16 (1788746656) [ 409.992365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.065312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.126574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 410.150648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 417.258628] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 22:04:24 (1788746664) [ 417.363319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.424830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.515369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 417.544177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 423.970763] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 22:04:31 (1788746671) [ 424.060426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.121445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.289739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 424.348706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 431.971298] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 22:04:38 (1788746678) [ 432.252166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.330298] LNet: 17437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.344879] LNet: 17437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 432.387484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.512367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 432.562789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 437.897956] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 22:04:45 (1788746685) [ 438.213068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.289255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.407994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 438.847050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.483667] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 22:04:53 (1788746693) [ 446.761984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.811168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.029711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.340979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.734656] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 22:05:01 (1788746701) [ 454.900111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.977818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.191840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.558475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.453331] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 22:05:09 (1788746709) [ 462.908351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.972587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.515387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.831908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.353778] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 22:05:17 (1788746717) [ 470.537542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.582716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.667188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.729581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.821470] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 22:05:24 (1788746724) [ 477.164472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.206988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.315450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.396950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 483.894072] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 22:05:31 (1788746731) [ 484.159100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.199696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.259933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 490.635233] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 22:05:37 (1788746737) [ 490.721396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.783097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.892248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 498.259673] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 22:05:45 (1788746745) [ 498.448704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.479535] LNet: 22874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 498.488023] LNet: 22874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 498.522188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.596993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 499.030273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 504.155789] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 22:05:51 (1788746751) [ 504.277702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.326722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.369889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 504.384781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 509.731620] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 22:05:57 (1788746757) [ 509.832967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.865933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.968142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 510.006479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 516.094478] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 22:06:03 (1788746763) [ 516.182976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.246846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.367732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 516.397868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 522.508486] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 22:06:09 (1788746769) [ 522.571852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.619058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.721383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 522.801670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 529.173599] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 22:06:16 (1788746776) [ 529.246142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.293918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.369518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 529.413558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 529.428390] LNetError: 25821:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 529.468976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 529.498317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 535.274715] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 22:06:22 (1788746782) [ 535.388596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.445406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.527207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.dmXujvxvHJ [ 535.589298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.dmXujvxvHJ [ 535.634535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.dmXujvxvHJ [ 540.761936] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 22:06:28 (1788746788) [ 541.030777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.064453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.121988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 541.152969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 541.188594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 546.442355] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 22:06:33 (1788746793) [ 546.730349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.806921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.889582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 546.917603] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 548.549930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 549.875879] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 551.903624] Key type lgssc registered [ 553.288091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 556.904732] LNetError: 27582:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.15@tcp99 has no available nets [ 559.288795] Key type lgssc unregistered [ 559.565972] LNetError: 28686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.615109] LNet: Removed LNI 192.168.201.15@tcp [ 561.206738] Key type .llcrypt unregistered [ 561.209283] Key type ._llcrypt unregistered [ 567.144094] Key type ._llcrypt registered [ 567.150629] Key type .llcrypt registered [ 568.205342] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 22:06:55 (1788746815) [ 568.820485] Key type .llcrypt unregistered [ 568.823872] Key type ._llcrypt unregistered [ 574.905509] Key type ._llcrypt registered [ 574.908526] Key type .llcrypt registered [ 575.022567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 575.034727] alg: No test for adler32 (adler32-zlib) [ 575.981305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.112567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 576.152710] LNet: 29596:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 576.168576] LNet: 29596:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 576.180770] LNet: 29596:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 576.208471] LNet: Message forwarding will be enabled in 2s [ 579.299472] LNet: Message forwarding enabled [ 579.838799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 579.858751] LNet: Message forwarding will stop in 2s to 6s [ 583.404959] LNet: Message forwarding disabled [ 584.575426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 584.602163] LNet: 29695:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 584.607662] LNet: 29695:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 584.615246] LNet: 29695:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 584.631945] LNet: Message forwarding will be enabled in 2s [ 584.868430] LNet: 29708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.564519] Key type .llcrypt unregistered [ 585.567420] Key type ._llcrypt unregistered [ 597.600244] Key type ._llcrypt registered [ 597.602029] Key type .llcrypt registered [ 599.125450] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 22:07:25 (1788746845) [ 599.512697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 599.540993] alg: No test for adler32 (adler32-zlib) [ 600.542029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.643496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 605.874307] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 22:07:33 (1788746853) [ 606.079147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.114394] LNet: 31106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.167928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.227192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 606.259155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 606.283146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 606.316069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 606.340510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 606.370801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 606.395931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 606.427295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 606.454766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 606.482682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 606.508864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 606.536829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 606.565081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 606.595651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 606.627125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 606.650805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 606.683181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 606.731792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 606.792617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 606.811566] LNetError: 31151:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 612.812806] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 22:07:39 (1788746859) [ 612.900811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.936640] LNet: 31717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.952623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.046881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 619.797966] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 22:07:46 (1788746866) [ 621.989944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg115-server [ 623.928504] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 22:07:51 (1788746871) [ 624.351232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.402163] LNet: 32653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.435337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.490400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 624.537326] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 625.434561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[16-20]@tcp [ 625.449989] LNet: 32787:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 625.513961] LNetError: 32657:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.16@tcp is being used as a gateway but routing feature is not turned on [ 625.524245] LNetError: 32657:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.16@tcp has gone from up to down [ 630.859168] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 22:07:58 (1788746878) [ 632.381319] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg115-server [ 634.495704] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 22:08:01 (1788746881) [ 634.641740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.669926] LNet: 33721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.678904] LNetError: 33721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 634.706725] LNet: Removed LNI 192.168.201.15@tcp [ 634.746478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.804841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 634.829431] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 635.591711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[16-21/2]@tcp [ 635.628363] LNet: 33839:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 635.640441] LNet: 33839:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 635.663468] LNetError: 33726:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.16@tcp is being used as a gateway but routing feature is not turned on [ 635.671360] LNetError: 33726:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 635.678158] LNetError: 33726:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.16@tcp has gone from up to down [ 635.678633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[16-21/2]@tcp [ 635.685872] LNetError: 33726:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 641.262132] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 22:08:08 (1788746888) [ 641.450308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.464735] LNet: 34431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.473678] LNetError: 34431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.487690] LNet: Removed LNI 192.168.201.15@tcp [ 641.509975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.614506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 641.701545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.764912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.873809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 641.954356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.011486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.074382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 642.161698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.198111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.262839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 642.368380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.433347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.494146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 642.592490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.625724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.718907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 642.837607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.901484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.010898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 643.092551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.134189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.192637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 643.254237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.287639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.343859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 643.430814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.472864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.589299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 643.679938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.712636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.752554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 643.808632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.862493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.952967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.784629] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 22:08:17 (1788746897) [ 651.707473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg115-server [ 653.038025] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 22:08:20 (1788746900) [ 654.927240] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg115-server [ 656.220447] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 22:08:23 (1788746903) [ 656.297640] LNet: 35826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.313325] LNet: 35826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 656.709899] Key type .llcrypt unregistered [ 656.712326] Key type ._llcrypt unregistered [ 661.201583] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 661.299339] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 661.301560] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 661.308204] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 663.833930] Key type ._llcrypt registered [ 663.836654] Key type .llcrypt registered [ 663.971358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 663.979250] alg: No test for adler32 (adler32-zlib) [ 664.908318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.966974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 665.008871] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 665.632378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 665.650738] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 667.116667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 667.131787] LNet: Removed LNI 10.1.2.3@tcp [ 667.186695] LNet: 36548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.189539] LNetError: 36548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.208387] LNet: Removed LNI 192.168.201.15@tcp [ 667.700516] Key type .llcrypt unregistered [ 667.705064] Key type ._llcrypt unregistered [ 677.128678] Key type ._llcrypt registered [ 677.133182] Key type .llcrypt registered [ 678.340745] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 22:08:45 (1788746925) [ 678.956791] Key type .llcrypt unregistered [ 678.968077] Key type ._llcrypt unregistered [ 685.355251] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 685.487411] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 685.490684] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 685.496207] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 688.071817] Key type ._llcrypt registered [ 688.074512] Key type .llcrypt registered [ 688.295608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 688.311827] alg: No test for adler32 (adler32-zlib) [ 689.207424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.289798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.354383] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 689.914787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 689.935791] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 691.997277] LNet: 38038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.004693] LNetError: 38038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 692.009771] LNet: Removed LNI 10.1.2.3@tcp [ 692.022925] LNet: Removed LNI 192.168.201.15@tcp [ 692.455828] Key type .llcrypt unregistered [ 692.457833] Key type ._llcrypt unregistered [ 702.906818] Key type ._llcrypt registered [ 702.911362] Key type .llcrypt registered [ 704.085257] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 22:09:11 (1788746951) [ 704.646333] Key type .llcrypt unregistered [ 704.648667] Key type ._llcrypt unregistered [ 711.039493] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 711.180745] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 711.183943] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 711.187689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 713.654164] Key type ._llcrypt registered [ 713.657292] Key type .llcrypt registered [ 713.764241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 713.775788] alg: No test for adler32 (adler32-zlib) [ 714.667643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.692134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 714.711510] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 716.354039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 716.371670] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 718.033270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 718.053727] LNet: Removed LNI 10.1.2.3@tcp [ 718.065321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 718.097266] LNet: Removed LNI 10.1.2.31@tcp [ 718.129379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 718.172106] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 719.016716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 719.035700] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 719.768400] LNet: 39952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.774855] LNetError: 39952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 719.781772] LNet: Removed LNI 10.1.2.31@tcp [ 719.791875] LNet: Removed LNI 10.1.2.3@tcp [ 720.390850] Key type .llcrypt unregistered [ 720.393660] Key type ._llcrypt unregistered [ 731.435591] Key type ._llcrypt registered [ 731.436926] Key type .llcrypt registered [ 732.464411] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 22:09:39 (1788746979) [ 732.805837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 732.820462] alg: No test for adler32 (adler32-zlib) [ 733.826674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.897186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.914983] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 734.577939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.593969] LNet: 40903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.598721] LNetError: 40903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.609772] LNet: Removed LNI 192.168.201.15@tcp [ 734.625465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.690077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.15@tcp [ 734.717539] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 735.290629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.15@tcp [ 735.315578] LNet: Removed LNI 192.168.201.15@tcp [ 735.380990] LNet: 41039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.843613] Key type .llcrypt unregistered [ 735.848739] Key type ._llcrypt unregistered [ 745.857496] Key type ._llcrypt registered [ 745.860882] Key type .llcrypt registered [ 747.112604] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 22:09:54 (1788746994) [ 749.053744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg115-server [ 751.148775] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 22:09:58 (1788746998) [ 751.649878] Key type .llcrypt unregistered [ 751.654757] Key type ._llcrypt unregistered [ 756.659129] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 757.018729] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 757.021605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 757.029345] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 759.544352] Key type ._llcrypt registered [ 759.546818] Key type .llcrypt registered [ 759.736367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.751092] alg: No test for adler32 (adler32-zlib) [ 760.784432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.866775] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 761.650614] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 763.314470] LNet: 42870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.319715] LNetError: 42870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.325602] LNet: Removed LNI 192.168.201.15@tcp1 [ 763.338860] LNet: Removed LNI 10.1.2.3@tcp2 [ 763.795763] Key type .llcrypt unregistered [ 763.798267] Key type ._llcrypt unregistered [ 774.182513] Key type ._llcrypt registered [ 774.184623] Key type .llcrypt registered [ 775.207776] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 22:10:22 (1788747022) [ 775.576522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 775.589982] alg: No test for adler32 (adler32-zlib) [ 776.520091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.568673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.587889] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 777.199375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.212915] LNet: 43837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.216267] LNetError: 43837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.226054] LNet: Removed LNI 192.168.201.15@tcp [ 777.239586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.286255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 777.302643] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 782.323965] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 22:10:29 (1788747029) [ 782.426661] LNet: 44420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.434569] LNetError: 44420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.455589] LNet: Removed LNI 192.168.201.15@tcp [ 782.954624] Key type .llcrypt unregistered [ 782.956784] Key type ._llcrypt unregistered [ 787.867351] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 788.037802] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 788.049202] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 788.071825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 790.643825] Key type ._llcrypt registered [ 790.646040] Key type .llcrypt registered [ 790.855825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.876363] alg: No test for adler32 (adler32-zlib) [ 791.784134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.858000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.886130] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 792.452387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 792.471058] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.003754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.019400] LNet: 45167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.023951] LNetError: 45167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 794.028539] LNet: Removed LNI 10.1.2.3@tcp [ 794.034856] LNet: Removed LNI 192.168.201.15@tcp [ 794.050834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.098826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 794.126452] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 794.129733] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.481776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.542706] LNet: 45194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.547959] LNetError: 45194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 794.552934] LNetError: 45194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 794.560176] LNet: Removed LNI 10.1.2.3@tcp [ 794.577537] LNet: Removed LNI 192.168.201.15@tcp [ 794.598905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.677632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 794.710316] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 794.717786] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.961322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 794.977182] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 795.759915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.791359] LNet: 45357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.804306] LNetError: 45357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 795.809942] LNetError: 45357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 795.816706] LNet: Removed LNI 10.1.2.3@tcp [ 795.819729] LNet: Removed LNI 192.168.201.15@tcp [ 795.831449] LNet: Removed LNI 192.168.201.15@tcp2 [ 795.851701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.902812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 795.925481] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 795.927872] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.931378] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 801.802971] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 22:10:49 (1788747049) [ 801.993756] LNet: 45953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.001026] LNetError: 45953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.012774] LNetError: 45953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 802.026737] LNet: Removed LNI 10.1.2.3@tcp [ 802.030594] LNet: Removed LNI 192.168.201.15@tcp [ 802.046848] LNet: Removed LNI 192.168.201.15@tcp2 [ 802.383804] Key type .llcrypt unregistered [ 802.385858] Key type ._llcrypt unregistered [ 807.467946] Key type ._llcrypt registered [ 807.469788] Key type .llcrypt registered [ 807.746332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.776497] alg: No test for adler32 (adler32-zlib) [ 808.742206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.808321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 808.837157] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 809.625371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.642517] LNet: 46345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.646573] LNetError: 46345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.661726] LNet: Removed LNI 192.168.201.15@tcp [ 809.681061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.793988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 809.841949] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 810.485468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.534039] LNet: 46375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.540621] LNetError: 46375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.553087] LNet: Removed LNI 192.168.201.15@tcp [ 810.624734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.725491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 810.749268] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 811.187432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.214476] LNet: Removed LNI 192.168.201.15@tcp [ 811.230412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.284334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 811.307452] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 811.692461] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 811.788048] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.790884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.796907] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 814.210982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.226778] LNet: 46483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.230703] LNet: 46483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 814.236161] LNetError: 46483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.238849] LNetError: 46483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 814.247529] LNet: Removed LNI 192.168.201.15@tcp [ 814.258383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.303811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 814.317441] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 819.468336] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 22:11:06 (1788747066) [ 819.579646] LNet: 47075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.585990] LNetError: 47075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.609617] LNet: Removed LNI 192.168.201.15@tcp [ 820.049848] Key type .llcrypt unregistered [ 820.051466] Key type ._llcrypt unregistered [ 825.398670] Key type ._llcrypt registered [ 825.406525] Key type .llcrypt registered [ 825.657108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 825.682964] alg: No test for adler32 (adler32-zlib) [ 826.646482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.701648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.741944] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 827.493830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.508527] LNet: 47475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.515774] LNetError: 47475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.527707] LNet: Removed LNI 192.168.201.15@tcp [ 827.542883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.589299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 827.606914] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 833.642871] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 22:11:21 (1788747081) [ 833.727091] LNet: 48060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.736798] LNetError: 48060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.759406] LNet: Removed LNI 192.168.201.15@tcp [ 834.226628] Key type .llcrypt unregistered [ 834.228343] Key type ._llcrypt unregistered [ 838.938708] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 839.125333] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 839.130565] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 839.133984] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 841.519582] Key type ._llcrypt registered [ 841.521564] Key type .llcrypt registered [ 841.647600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 841.661423] alg: No test for adler32 (adler32-zlib) [ 842.577550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.622785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 842.660343] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 843.271043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.291634] LNet: 48514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.296870] LNetError: 48514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.313237] LNet: Removed LNI 192.168.201.15@tcp [ 843.328064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.376671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 843.397317] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 848.453096] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 22:11:36 (1788747096) [ 848.621851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.643354] LNet: 49112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.650095] LNetError: 49112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.668247] LNet: Removed LNI 192.168.201.15@tcp [ 848.686641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.781915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 852.973325] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 22:11:40 (1788747100) [ 853.252190] LNet: 49694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.830719] Key type .llcrypt unregistered [ 853.832431] Key type ._llcrypt unregistered [ 858.661908] Key type ._llcrypt registered [ 858.664579] Key type .llcrypt registered [ 858.904804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.930186] alg: No test for adler32 (adler32-zlib) [ 859.925049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.993952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.016064] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 860.777980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.794194] LNet: 50088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.798945] LNetError: 50088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.812156] LNet: Removed LNI 192.168.201.15@tcp [ 860.853335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.947662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 860.997091] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 867.079584] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 22:11:54 (1788747114) [ 867.170931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.192314] LNet: 50675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.197199] LNetError: 50675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.216415] LNet: Removed LNI 192.168.201.15@tcp [ 867.247256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.328540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 872.818515] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 22:12:00 (1788747120) [ 874.334935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 875.548716] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 22:12:03 (1788747123) [ 876.800266] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 878.058911] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 22:12:05 (1788747125) [ 879.157533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 880.596121] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 22:12:07 (1788747127) [ 881.941148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 883.695430] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 22:12:10 (1788747130) [ 884.986544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 886.360136] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 22:12:13 (1788747133) [ 886.422364] LNet: 52694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.909657] Key type .llcrypt unregistered [ 886.913517] Key type ._llcrypt unregistered [ 891.296641] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.429789] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.433238] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.439317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 893.872136] Key type ._llcrypt registered [ 893.873843] Key type .llcrypt registered [ 894.068740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.086895] alg: No test for adler32 (adler32-zlib) [ 895.058857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.160128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 895.193820] LNet: Added LNI 192.168.201.15@tcp [8/512/0/180] [ 895.201637] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 895.206155] LNet: 53035:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 895.312599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.334474] LNet: 53054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.339103] LNetError: 53054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.342991] LNet: Removed LNI 10.1.2.3@tcp [ 895.344947] LNetError: 53054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.357959] LNet: Removed LNI 192.168.201.15@tcp [ 895.380800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.438618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 895.466895] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 895.530272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.544578] LNet: 53078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.547427] LNetError: 53078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.553793] LNet: Removed LNI 192.168.201.15@tcp [ 895.563210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.606648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 895.625809] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 895.628355] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.629856] LNet: 53086:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 895.731982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.749949] LNet: 53104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.754665] LNetError: 53104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.758062] LNet: Removed LNI 10.1.2.3@tcp [ 895.759536] LNetError: 53104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.771792] LNet: Removed LNI 192.168.201.15@tcp [ 895.796376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.871555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 895.887369] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 895.889440] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.890986] LNet: 53112:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 896.032372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.047325] LNet: 53138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.052160] LNetError: 53138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 896.057168] LNet: Removed LNI 10.1.2.3@tcp [ 896.059933] LNetError: 53138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.074861] LNet: Removed LNI 192.168.201.15@tcp [ 896.090247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.135194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 896.157838] LNet: Added LNI 192.168.201.15@tcp [8/256/0/182] [ 896.162154] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 896.245261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.261791] LNet: 53166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.267791] LNetError: 53166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 896.272537] LNet: Removed LNI 10.1.2.3@tcp [ 896.274437] LNetError: 53166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.283907] LNet: Removed LNI 192.168.201.15@tcp [ 896.297518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.350239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 896.368760] LNet: Added LNI 192.168.201.15@tcp [8/256/0/182] [ 896.375698] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 896.378266] LNet: 53174:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 896.439798] LNet: Removed LNI 10.1.2.3@tcp [ 896.451326] LNet: Removed LNI 192.168.201.15@tcp [ 896.930873] Key type .llcrypt unregistered [ 896.936846] Key type ._llcrypt unregistered [ 906.322186] Key type ._llcrypt registered [ 906.324579] Key type .llcrypt registered [ 907.356306] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 22:12:34 (1788747154) [ 908.527447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 909.924589] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 22:12:37 (1788747157) [ 910.390575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 910.400590] alg: No test for adler32 (adler32-zlib) [ 911.316300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.366966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.387642] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 911.897740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.15 [ 911.914342] LNet: 54436:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 911.923634] LNet: Removed LNI 192.168.201.15@tcp [ 916.748594] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 22:12:44 (1788747164) [ 916.993090] LNet: 55006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.998849] LNetError: 55006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.011409] LNet: Removed LNI 192.168.201.15@tcp [ 917.420326] Key type .llcrypt unregistered [ 917.421478] Key type ._llcrypt unregistered [ 922.575387] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 922.855847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 925.459435] Key type ._llcrypt registered [ 925.461838] Key type .llcrypt registered [ 925.592559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.602634] alg: No test for adler32 (adler32-zlib) [ 926.812613] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 936.182329] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 22:13:03 (1788747183) [ 936.327900] LNet: 55911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.335289] LNetError: 55911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 936.354708] LNet: Removed LNI 10.1.2.3@tcp [ 936.926603] Key type .llcrypt unregistered [ 936.928394] Key type ._llcrypt unregistered [ 942.792095] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 943.067466] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 943.069951] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 943.075324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 945.683724] Key type ._llcrypt registered [ 945.686270] Key type .llcrypt registered [ 945.816352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.830895] alg: No test for adler32 (adler32-zlib) [ 946.725461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.656447] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 22:13:20 (1788747200) [ 955.525314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 958.150454] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 22:13:24 (1788747204) [ 958.308968] LNet: 57094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.316796] LNetError: 57094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 958.332183] LNet: Removed LNI 10.1.2.3@tcp [ 959.624649] Key type .llcrypt unregistered [ 959.631069] Key type ._llcrypt unregistered [ 969.702076] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 970.128638] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 970.132566] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 970.140049] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 972.068374] Key type ._llcrypt registered [ 972.070856] Key type .llcrypt registered [ 972.498609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.549635] alg: No test for adler32 (adler32-zlib) [ 973.797749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.978911] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 989.928544] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 22:13:56 (1788747236) [ 990.932707] LNet: 58322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.950650] LNetError: 58322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 990.969459] LNet: Removed LNI 10.1.2.3@tcp [ 992.251720] Key type .llcrypt unregistered [ 992.254720] Key type ._llcrypt unregistered [ 1000.551241] Key type ._llcrypt registered [ 1000.553255] Key type .llcrypt registered [ 1000.935343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1000.980092] alg: No test for adler32 (adler32-zlib) [ 1002.042284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1002.062835] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1012.264966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 1013.011203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_interrupt -r 1 [ 1013.041767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_interrupt -r 1 [ 1013.066541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1013.084519] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1013.280766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1013.312278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1013.341106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1013.343446] LNet: 1 local NIs in recovery (showing 1): 192.168.201.15@tcp [ 1013.371567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1013.397702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1013.424828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1013.458784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1013.485567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1013.510862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1013.527718] LNet: 58759:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1013.546608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1013.598054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1013.658122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1013.720756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1013.787493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1013.839696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1013.895923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1014.416424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_dropped -r 1 [ 1014.451915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_dropped -r 1 [ 1014.480625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1014.496010] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1014.677522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1014.713470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1014.756236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1014.798665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1014.830086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1014.851332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1014.869983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1014.894779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1014.917189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1014.933494] LNet: 58871:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1014.950121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1014.972750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1015.000597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1015.024564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1015.049455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1015.079794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.127183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1015.601695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_aborted -r 1 [ 1015.657771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_aborted -r 1 [ 1015.713528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1015.739604] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1015.958276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.989367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1016.019213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.045678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.075506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1016.103447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.136632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.163685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.185365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.200624] LNet: 58982:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.216736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.239965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.265897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.292475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.319378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.347292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.375838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1016.852590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_no_route -r 1 [ 1016.884426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_no_route -r 1 [ 1016.919639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1017.106216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.127645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1017.144934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.175490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.210756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1017.240635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.288789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.345316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.386268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.425874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.464635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.493628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.540033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.575030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.599906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1017.624829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1018.118606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_timeout -r 1 [ 1018.148212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_timeout -r 1 [ 1018.188635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1018.209762] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1018.213254] LNet: Skipped 2 previous similar messages [ 1018.400632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1018.426962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1018.453535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1018.481868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1018.507785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1018.542600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1018.575293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1018.609429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.653398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.680923] LNet: 59204:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1018.685219] LNet: 59204:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1018.706608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.747171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.791126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.849314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.886470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.924608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.977578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1019.602967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_error -r 1 [ 1019.632863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_error -r 1 [ 1019.661754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1019.941477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.981550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1020.017684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.048697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.081819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1020.110730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.140877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.172528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.209933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.258434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.298788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.349343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.393442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.439092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1020.500874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1020.579832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1020.879717] LNet: 59356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.885873] LNetError: 59356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.901265] LNet: Removed LNI 192.168.201.15@tcp [ 1021.356481] Key type .llcrypt unregistered [ 1021.358180] Key type ._llcrypt unregistered [ 1040.526982] Key type ._llcrypt registered [ 1040.534245] Key type .llcrypt registered [ 1041.976850] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 22:14:48 (1788747288) [ 1042.963283] Key type .llcrypt unregistered [ 1042.964902] Key type ._llcrypt unregistered [ 1050.635512] Key type ._llcrypt registered [ 1050.640532] Key type .llcrypt registered [ 1050.965807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.993431] alg: No test for adler32 (adler32-zlib) [ 1051.979282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1052.051393] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1059.579693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 1069.752181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.794698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1069.817914] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1071.386830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_interrupt -r 1 [ 1071.413237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_interrupt -r 1 [ 1071.462780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_interrupt -r 1 [ 1071.526985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_interrupt -r 1 [ 1071.569746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_interrupt -r 1 [ 1071.602536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_interrupt -r 1 [ 1071.671208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_interrupt -r 1 [ 1071.707856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_interrupt -r 1 [ 1071.745675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1071.766789] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1071.790739] LNet: 1 local NIs in recovery (showing 1): 192.168.201.15@tcp [ 1072.113711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.138953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1072.189268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.239644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1072.289953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1072.329578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1072.367700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1072.397907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.422274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.436327] LNet: 60830:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.449851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.473779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.495442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.530867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.569044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.607849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.635716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1073.022168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_dropped -r 1 [ 1073.048594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_dropped -r 1 [ 1073.084170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_dropped -r 1 [ 1073.113843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_dropped -r 1 [ 1073.145445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_dropped -r 1 [ 1073.185661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_dropped -r 1 [ 1073.253282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_dropped -r 1 [ 1073.292730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_dropped -r 1 [ 1073.331222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1073.353483] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1073.357927] LNet: Skipped 2 previous similar messages [ 1073.663926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1073.708808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1073.738898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1073.768479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1073.813233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1073.855230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1073.881224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1073.913689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1073.950770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1073.979932] LNet: 60953:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.009955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.052284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.085881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.122558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.146594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.183814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.248723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1074.886952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_aborted -r 1 [ 1074.947930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_aborted -r 1 [ 1075.000378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_aborted -r 1 [ 1075.065221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_aborted -r 1 [ 1075.113851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_aborted -r 1 [ 1075.189125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_aborted -r 1 [ 1075.235510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_aborted -r 1 [ 1075.297261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_aborted -r 1 [ 1075.348924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1075.392922] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1075.400157] LNet: Skipped 2 previous similar messages [ 1075.680981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.716730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1075.744743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.794768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.831843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1075.887411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.941294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.991993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.038190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.065968] LNet: 61077:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1076.095835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1076.141865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1076.169846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1076.206793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1076.243763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.277318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1076.363875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1076.933302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_no_route -r 1 [ 1077.010000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_no_route -r 1 [ 1077.064655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_no_route -r 1 [ 1077.110613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_no_route -r 1 [ 1077.177618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_no_route -r 1 [ 1077.237862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_no_route -r 1 [ 1077.279993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_no_route -r 1 [ 1077.345579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_no_route -r 1 [ 1077.428205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1077.462716] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1077.474482] LNet: Skipped 2 previous similar messages [ 1077.919107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1077.969566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1078.019967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1078.069694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1078.113513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1078.151358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1078.189285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1078.222745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1078.262479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1078.284803] LNet: 61200:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1078.299850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.329427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1078.357533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1078.393666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1078.442966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.481766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1078.549123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1079.148750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_timeout -r 1 [ 1079.201342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_timeout -r 1 [ 1079.249077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_timeout -r 1 [ 1079.297223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_timeout -r 1 [ 1079.328647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_timeout -r 1 [ 1079.365118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_timeout -r 1 [ 1079.401808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_timeout -r 1 [ 1079.456821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_timeout -r 1 [ 1079.512589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1079.803308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.844814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1079.893632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.924350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.961317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1079.995431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.030672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.060638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.084417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1080.112913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1080.140476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1080.168554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1080.209585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1080.325330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1080.387414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1080.443409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1081.148596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_error -r 1 [ 1081.233995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_error -r 1 [ 1081.339383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_error -r 1 [ 1081.382727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_error -r 1 [ 1081.434835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_error -r 1 [ 1081.471388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_error -r 1 [ 1081.530740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_error -r 1 [ 1081.597845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_error -r 1 [ 1081.654595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1081.698574] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1081.701882] LNet: Skipped 7 previous similar messages [ 1082.049712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.095476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1082.123914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.152395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.188984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1082.236048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.277652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1082.317896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1082.349978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1082.366137] LNet: 61447:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1082.372114] LNet: 61447:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1082.400078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1082.427516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1082.468681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.508101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.544623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.585799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.654862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1088.353945] LNet: 61522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.366163] LNetError: 61522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.371593] LNet: Removed LNI 192.168.201.15@tcp [ 1089.456552] LNet: Removed LNI 192.168.201.15@tcp1 [ 1089.989965] Key type .llcrypt unregistered [ 1089.994054] Key type ._llcrypt unregistered [ 1111.999225] Key type ._llcrypt registered [ 1112.005121] Key type .llcrypt registered [ 1113.509241] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:16:00 (1788747360) [ 1115.110430] Key type .llcrypt unregistered [ 1115.112253] Key type ._llcrypt unregistered [ 1122.864340] Key type ._llcrypt registered [ 1122.871191] Key type .llcrypt registered [ 1123.273567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.311227] alg: No test for adler32 (adler32-zlib) [ 1124.516804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1124.565643] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1133.870348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 1134.698983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e remote_dropped -r 1 [ 1134.728364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e remote_dropped -r 1 [ 1134.755442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1134.770647] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1135.007498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1135.031758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1135.093823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1135.130068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1135.183469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1135.232162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1135.270438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.328485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.360045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.412400] LNet: 62770:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1135.447934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.504549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.567470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.625702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.677217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.724833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.792297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1136.462789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e remote_error -r 1 [ 1136.527781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e remote_error -r 1 [ 1136.587466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1136.624631] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1136.909719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.960787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.005177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.051836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.095887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1137.128429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.172699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.217203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.259493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.276608] LNet: 62882:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.296683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.332976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.363831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.411355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.463883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.539633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.610581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1138.327855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e remote_timeout -r 1 [ 1138.377376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e remote_timeout -r 1 [ 1138.428725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1138.478971] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1138.791961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.850906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1138.901806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.961946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.009203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1139.058795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.089869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.117328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.148311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.163932] LNet: 62993:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.181359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.209681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.285852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.314664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.347402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.387942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.440762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1139.671893] LNet: 63034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.675935] LNetError: 63034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.688396] LNet: Removed LNI 192.168.201.15@tcp [ 1140.091418] Key type .llcrypt unregistered [ 1140.093747] Key type ._llcrypt unregistered [ 1157.978781] Key type ._llcrypt registered [ 1157.980802] Key type .llcrypt registered [ 1159.380390] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:16:46 (1788747406) [ 1160.428431] Key type .llcrypt unregistered [ 1160.431404] Key type ._llcrypt unregistered [ 1167.774948] Key type ._llcrypt registered [ 1167.776712] Key type .llcrypt registered [ 1168.008102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1168.049261] alg: No test for adler32 (adler32-zlib) [ 1169.073599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1169.107122] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1177.801960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 1189.455145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1189.498138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1189.525568] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1191.015425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e remote_dropped -r 1 [ 1191.050542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e remote_dropped -r 1 [ 1191.084512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e remote_dropped -r 1 [ 1191.130144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e remote_dropped -r 1 [ 1191.199844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e remote_dropped -r 1 [ 1191.257806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e remote_dropped -r 1 [ 1191.307227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e remote_dropped -r 1 [ 1191.364955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e remote_dropped -r 1 [ 1191.436167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1191.462236] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1191.468991] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.115@tcp [ 1191.712710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.750227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1191.782514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.833993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.911519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1191.989532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.023659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.082297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.114736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.132430] LNet: 64499:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.145793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.196814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.245924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.286240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.330176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.353831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.392268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1192.442858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.098842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e remote_error -r 1 [ 1193.128595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e remote_error -r 1 [ 1193.163805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e remote_error -r 1 [ 1193.200388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e remote_error -r 1 [ 1193.234755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e remote_error -r 1 [ 1193.271700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e remote_error -r 1 [ 1193.308774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e remote_error -r 1 [ 1193.367505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e remote_error -r 1 [ 1193.417996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1193.435678] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1193.440766] LNet: Skipped 2 previous similar messages [ 1193.686248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.720957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.759136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.795955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.836918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1193.871779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.900691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.931475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.977083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.992204] LNet: 64624:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1194.012133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.044960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.081470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.131883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.172827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.210141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.271907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1194.305383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.084654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e remote_timeout -r 1 [ 1195.136850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e remote_timeout -r 1 [ 1195.181881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e remote_timeout -r 1 [ 1195.255732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e remote_timeout -r 1 [ 1195.313378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e remote_timeout -r 1 [ 1195.350623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e remote_timeout -r 1 [ 1195.413960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e remote_timeout -r 1 [ 1195.484396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e remote_timeout -r 1 [ 1195.535819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.115@tcp [ 1195.558353] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1195.909780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1195.958430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1196.046351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.116377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.173675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1196.196296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.228853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.278290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.304543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.328501] LNet: 64750:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.344646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.375988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.417789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.464453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.513357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.565683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.609956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1196.658514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.275362] LNet: 64827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.287607] LNetError: 64827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.294715] LNet: Removed LNI 192.168.201.15@tcp [ 1202.322179] LNet: Removed LNI 192.168.201.15@tcp1 [ 1202.941068] Key type .llcrypt unregistered [ 1202.943148] Key type ._llcrypt unregistered [ 1222.234918] Key type ._llcrypt registered [ 1222.236802] Key type .llcrypt registered [ 1223.784058] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:17:50 (1788747470) [ 1225.948573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1227.677087] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:17:54 (1788747474) [ 1228.611418] Key type .llcrypt unregistered [ 1228.613205] Key type ._llcrypt unregistered [ 1236.583322] Key type ._llcrypt registered [ 1236.587726] Key type .llcrypt registered [ 1236.894122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1236.925715] alg: No test for adler32 (adler32-zlib) [ 1238.009295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1238.035379] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1247.493397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 1248.583617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e network_timeout -r 1 [ 1248.655612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e network_timeout -r 1 [ 1248.708712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 1248.734545] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1249.050817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.104098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1249.187125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.240382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.329795] LNet: 1 local NIs in recovery (showing 1): 192.168.201.15@tcp [ 1249.333246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1249.341823] LNet: There was an unexpected network error while writing to 192.168.201.15: rc = -22 [ 1249.446994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.523678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.559885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.609356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.641182] LNet: 66369:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.688184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.762629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.805640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.875732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.958806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.051794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.363833] LNet: 66408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.374723] LNetError: 66408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.432774] LNet: Removed LNI 192.168.201.15@tcp [ 1252.023689] Key type .llcrypt unregistered [ 1252.028175] Key type ._llcrypt unregistered [ 1272.128825] Key type ._llcrypt registered [ 1272.130878] Key type .llcrypt registered [ 1272.419372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1272.448046] alg: No test for adler32 (adler32-zlib) [ 1273.335122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1273.355163] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1280.933068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 1290.913879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1290.952323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1290.988296] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1292.140556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e network_timeout -r 1 [ 1292.173495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e network_timeout -r 1 [ 1292.205655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e network_timeout -r 1 [ 1292.237213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e network_timeout -r 1 [ 1292.261733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e network_timeout -r 1 [ 1292.286413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e network_timeout -r 1 [ 1292.316290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e network_timeout -r 1 [ 1292.348947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e network_timeout -r 1 [ 1292.376712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 1292.395268] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1292.602781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.626422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1292.652648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.680798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.709812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1292.737974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.766274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.795206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.822499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1292.839423] LNet: 1 local NIs in recovery (showing 1): 192.168.201.15@tcp [ 1292.848494] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.115@tcp [ 1292.855344] LNet: 67284:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1292.871388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1292.906816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1292.933797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1292.960919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1292.999988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.042514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.783197] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1298.181677] LNet: 67358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.190902] LNetError: 67358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.198127] LNet: Removed LNI 192.168.201.15@tcp [ 1298.211456] LNet: Removed LNI 192.168.201.15@tcp1 [ 1298.755679] Key type .llcrypt unregistered [ 1298.764678] Key type ._llcrypt unregistered [ 1317.377574] Key type ._llcrypt registered [ 1317.379637] Key type .llcrypt registered [ 1318.783615] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 22:19:25 (1788747565) [ 1319.199912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1319.229886] alg: No test for adler32 (adler32-zlib) [ 1320.189391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1320.242243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1320.268977] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1321.172913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1321.198781] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1322.453138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 1322.590224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1322.692046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1322.768778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1322.841979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1322.920984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1322.962780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.15@tcp [ 1322.979641] LNetError: 68454:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.15@tcp to recovery [ 1323.295373] LNet: 1 local NIs in recovery (showing 1): 192.168.201.15@tcp [ 1323.296387] LNet: There was an unexpected network error while writing to 192.168.201.15: rc = -22 [ 1325.343659] LNet: There was an unexpected network error while writing to 192.168.201.15: rc = -22 [ 1326.636533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1329.439428] LNet: There was an unexpected network error while writing to 192.168.201.15: rc = -22 [ 1330.367316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1330.418879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1330.495798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1330.542337] LNet: 68561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.547053] LNetError: 68561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.558215] LNet: Removed LNI 192.168.201.15@tcp [ 1330.573313] LNet: Removed LNI 192.168.201.15@tcp1 [ 1330.610799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.719441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1330.769751] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1331.661570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1331.685833] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1332.655778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 1332.715417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1332.808400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1332.879276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1332.924773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1332.986596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1333.052734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1333.129360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.15@tcp [ 1333.174674] LNetError: 68834:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.15@tcp to recovery [ 1333.728548] LNet: There was an unexpected network error while writing to 192.168.201.15: rc = -22 [ 1336.861980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1339.872470] LNet: There was an unexpected network error while writing to 192.168.201.15: rc = -22 [ 1339.889340] LNet: Skipped 1 previous similar message [ 1345.179188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1345.206471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1345.253205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1353.589596] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 22:20:00 (1788747600) [ 1353.888942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1353.923504] LNet: 69553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.944471] LNetError: 69553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.957949] LNetError: 69553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1353.968297] LNet: Removed LNI 192.168.201.15@tcp [ 1353.982405] LNet: Removed LNI 192.168.201.15@tcp1 [ 1354.010775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1354.086458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1354.111494] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1354.767878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1354.783758] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1355.602909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 1355.659184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1355.712969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1355.775668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1355.837890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1355.889981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1355.950234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.15@tcp --health 0 [ 1356.129685] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.15@tcp [ 1356.148748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1357.151521] LNet: There was an unexpected network error while writing to 192.168.201.15: rc = -22 [ 1357.155973] LNet: Skipped 1 previous similar message [ 1372.149482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1372.373337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1372.412134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1372.448821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.15@tcp --health 500 [ 1372.593803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1379.468550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1379.553804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.586111] LNet: 70044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.604630] LNetError: 70044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.612591] LNetError: 70044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1379.626724] LNet: Removed LNI 192.168.201.15@tcp [ 1379.645746] LNet: Removed LNI 192.168.201.15@tcp1 [ 1379.685410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.782541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1379.846787] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1380.902690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1380.924884] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1381.852890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 1381.926103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.008599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1382.066413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1382.109912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1382.168689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1382.228035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1382.273222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.15@tcp --health 0 [ 1383.839912] LNet: There was an unexpected network error while writing to 192.168.201.15: rc = -22 [ 1383.846535] LNet: Skipped 3 previous similar messages [ 1384.915340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1394.261639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1394.343434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1394.428697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1403.229150] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 22:20:50 (1788747650) [ 1403.785907] LNet: 71092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.791476] LNetError: 71092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.799323] LNetError: 71092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1403.805368] LNet: Removed LNI 192.168.201.15@tcp [ 1403.818316] LNet: Removed LNI 192.168.201.15@tcp1 [ 1404.499883] Key type .llcrypt unregistered [ 1404.506800] Key type ._llcrypt unregistered [ 1412.412260] Key type ._llcrypt registered [ 1412.417704] Key type .llcrypt registered [ 1412.700716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1412.734884] alg: No test for adler32 (adler32-zlib) [ 1413.755317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1413.812256] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1425.165835] Lustre: DEBUG MARKER: Initial discovery [ 1425.240384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.115@tcp [ 1428.493525] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1428.576298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1428.653352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.115@tcp [ 1428.684396] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1428.783146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1428.894654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.115@tcp -r 1 -m GET -l 3 [ 1428.941129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.115@tcp -r 1 -m GET -e local_error [ 1428.983761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.115@tcp -r 1 -m PUT -l 6 [ 1429.216272] LNet: 1 local NIs in recovery (showing 1): 192.168.201.15@tcp [ 1430.575520] Lustre: DEBUG MARKER: Force oleg115-client.virtnet to discover 192.168.201.115@tcp (in background) [ 1430.645436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.115@tcp [ 1432.104740] Lustre: DEBUG MARKER: Force oleg115-server to discover 192.168.201.15@tcp [ 1433.706205] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1434.900053] Lustre: DEBUG MARKER: Wait for 71843 [ 1436.622803] Lustre: DEBUG MARKER: Finished wait on 71843 [ 1436.752814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1436.820799] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1436.831221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1437.184347] LNet: 72303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.189453] LNetError: 72303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.207402] LNet: Removed LNI 192.168.201.15@tcp [ 1437.738765] Key type .llcrypt unregistered [ 1437.741803] Key type ._llcrypt unregistered [ 1457.167353] Key type ._llcrypt registered [ 1457.174226] Key type .llcrypt registered [ 1459.152778] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:21:45 (1788747705) [ 1459.931486] Key type .llcrypt unregistered [ 1459.934227] Key type ._llcrypt unregistered [ 1466.913477] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1467.208791] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1467.219019] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1467.223440] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1469.910103] Key type ._llcrypt registered [ 1469.911760] Key type .llcrypt registered [ 1470.240464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.259853] alg: No test for adler32 (adler32-zlib) [ 1471.224820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.291390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.316894] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1472.318431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1472.355171] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1481.551511] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:22:08 (1788747728) [ 1482.169466] LNet: 74376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.174960] LNetError: 74376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1482.177793] LNet: Removed LNI 10.1.2.3@tcp [ 1482.196564] LNet: Removed LNI 192.168.201.15@tcp [ 1483.180346] Key type .llcrypt unregistered [ 1483.183934] Key type ._llcrypt unregistered [ 1492.007810] Key type ._llcrypt registered [ 1492.012983] Key type .llcrypt registered [ 1492.246466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1492.267900] alg: No test for adler32 (adler32-zlib) [ 1493.167127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.226348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.251832] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1494.086585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1494.115704] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1495.232415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.15@tcp [ 1496.351246] LNetError: Unexpected error -22 connecting to 192.168.201.15@tcp at host 192.168.201.15:988 [ 1502.867153] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:22:29 (1788747749) [ 1502.978941] LNet: 75497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.000170] LNetError: 75497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1503.010516] LNet: Removed LNI 10.1.2.3@tcp [ 1504.039512] LNet: Removed LNI 192.168.201.15@tcp [ 1504.779555] Key type .llcrypt unregistered [ 1504.781472] Key type ._llcrypt unregistered [ 1515.189262] Key type ._llcrypt registered [ 1515.192308] Key type .llcrypt registered [ 1515.492736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1515.526340] alg: No test for adler32 (adler32-zlib) [ 1516.390072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.415563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1516.429798] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1517.034964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1517.060060] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 1518.396326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.15@tcp1 --nid 192.168.201.15@tcp2 [ 1524.488810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.15@tcp1 192.168.201.15@tcp2 [ 1524.511335] LNetError: 76341:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.15@tcp1 to 192.168.201.15@tcp2 - Route table may be misconfigured [ 1524.528233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.15@tcp2 192.168.201.15@tcp1 [ 1538.089137] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:23:04 (1788747784) [ 1538.251950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.293144] LNet: 76908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.303586] LNetError: 76908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.318047] LNet: Removed LNI 192.168.201.15@tcp1 [ 1539.372921] LNet: Removed LNI 192.168.201.15@tcp2 [ 1539.503034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.717581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.788250] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1541.210904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1541.276172] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1542.990728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 1543.094626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.15@tcp 192.168.201.15@tcp [ 1543.170595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.15@tcp1 192.168.201.15@tcp1 [ 1543.278764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.15@tcp -d 192.168.201.15@tcp -e network_timeout [ 1543.407112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -e network_timeout [ 1543.542471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -e network_timeout [ 1543.622937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -e network_timeout [ 1543.765703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1543.842819] LNet: There was an unexpected network error while writing to 192.168.201.15: rc = -22 [ 1543.884123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1544.137844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1544.345200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1544.748844] LNet: 1 local NIs in recovery (showing 1): 192.168.201.15@tcp [ 1552.911249] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:23:20 (1788747800) [ 1553.317451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.390344] LNet: 77779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.406477] LNetError: 77779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.418455] LNetError: 77779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1553.428943] LNet: Removed LNI 192.168.201.15@tcp [ 1554.471809] LNet: Removed LNI 192.168.201.15@tcp1 [ 1554.601021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.864806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1555.391075] LNet: 77817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.964715] Key type .llcrypt unregistered [ 1555.967833] Key type ._llcrypt unregistered [ 1569.415302] Key type ._llcrypt registered [ 1569.417331] Key type .llcrypt registered [ 1571.513689] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:23:37 (1788747817) [ 1572.067911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1572.091211] alg: No test for adler32 (adler32-zlib) [ 1573.285249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.266921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1577.827781] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:23:44 (1788747824) [ 1578.031356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.081355] LNet: 78909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.150545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.254557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.304582] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1579.594706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1579.645113] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1581.060461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1581.202952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp1 [ 1581.346823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp1 [ 1590.323620] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:23:57 (1788747837) [ 1592.420711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg115-server [ 1594.028140] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:24:01 (1788747841) [ 1595.343845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg115-server [ 1597.346503] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:24:04 (1788747844) [ 1599.626196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg115-server [ 1601.616948] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:24:08 (1788747848) [ 1603.854268] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg115-server [ 1605.719546] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:24:12 (1788747852) [ 1607.759679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg115-server [ 1609.794386] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:24:16 (1788747856) [ 1611.756846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg115-server [ 1614.400944] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:24:20 (1788747860) [ 1616.653309] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg115-server [ 1618.612814] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:24:25 (1788747865) [ 1620.533285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg115-server [ 1622.068158] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:24:29 (1788747869) [ 1624.116724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg115-server [ 1625.825872] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:24:33 (1788747873) [ 1626.140665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.178644] LNet: 82791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.185447] LNetError: 82791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.190452] LNet: Removed LNI 192.168.201.15@tcp [ 1627.248403] LNet: Removed LNI 192.168.201.15@tcp1 [ 1627.320550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.415457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1627.466892] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1628.373169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1628.487697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1628.672813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.694752] LNet: 82938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.701427] LNetError: 82938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.705925] LNetError: 82938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1628.718420] LNet: Removed LNI 192.168.201.15@tcp [ 1628.738773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.790890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1628.810649] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1629.360199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1629.488230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.570665] LNet: Removed LNI 192.168.201.15@tcp [ 1630.643810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.732578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.786992] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1631.618637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1631.709733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1631.934606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.973389] LNet: 83231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.976104] LNet: 83231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1631.991546] LNetError: 83231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1632.000949] LNetError: 83231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1632.012528] LNet: Removed LNI 192.168.201.15@tcp [ 1632.045029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.120068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1632.173553] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1632.957197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1633.174622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.213463] LNet: Removed LNI 192.168.201.15@tcp [ 1634.237259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.295926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1634.326396] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1635.619738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1635.724901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1635.875968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.899614] LNetError: 83523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.906937] LNetError: 83523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1635.920338] LNet: Removed LNI 192.168.201.15@tcp [ 1635.942586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.019643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1636.050687] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1636.945485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1637.141122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.171390] LNet: 83669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.178210] LNet: 83669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1638.257834] LNet: Removed LNI 192.168.201.15@tcp [ 1638.293854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.329160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1638.352212] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1639.185326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1639.287167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1639.437346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.489113] LNet: Removed LNI 192.168.201.15@tcp [ 1640.519711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.626834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1640.663227] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1641.665556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1641.990197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.011480] LNetError: 83961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.018613] LNetError: 83961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1642.036289] LNet: Removed LNI 192.168.201.15@tcp [ 1642.088188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.182822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.209840] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1643.174842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1643.293432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1643.528472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.593516] LNet: Removed LNI 192.168.201.15@tcp [ 1644.662665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.728524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1644.765060] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1645.533220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1645.728488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.748688] LNet: 84254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.753754] LNet: 84254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1646.822904] LNet: Removed LNI 192.168.201.15@tcp [ 1646.867944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.939247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.972157] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1647.586930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1647.663992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1647.878253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.938362] LNet: Removed LNI 192.168.201.15@tcp [ 1648.983770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.054951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1649.111344] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1649.928278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1650.169709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.207866] LNetError: 84546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1650.212909] LNetError: 84546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1651.239432] LNet: Removed LNI 192.168.201.15@tcp [ 1651.278776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.379940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1651.429509] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1652.402475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1652.466168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.503899] LNet: Removed LNI 192.168.201.15@tcp [ 1652.520595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.576103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1652.597989] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1653.325226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1653.384898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.416261] LNet: Removed LNI 192.168.201.15@tcp [ 1653.432769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.494225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1653.571973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.605260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.664368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1653.745729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.777292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.820776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1653.909939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.974775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.048822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1654.119674] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1654.965218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.992261] LNet: Removed LNI 192.168.201.15@tcp [ 1655.006733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.095225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1662.056472] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:25:09 (1788747909) [ 1662.409061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.441955] LNet: 85567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.444148] LNet: 85567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1662.475380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.538782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1662.564599] LNet: Added LNI 192.168.201.15@tcp231 [8/256/0/180] [ 1664.408141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.464668] LNet: Removed LNI 192.168.201.15@tcp231 [ 1664.496180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.569723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1664.589912] LNet: Added LNI 192.168.201.15@tcp231 [8/256/0/180] [ 1664.594911] LNet: 85714:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1665.110920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1665.161656] LNet: Removed LNI 192.168.201.15@tcp231 [ 1665.203099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1665.269526] LNet: Added LNI 192.168.201.15@tcp231 [8/256/0/180] [ 1666.967779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.992668] LNetError: 85858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.997967] LNetError: 85858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1667.013988] LNet: Removed LNI 192.168.201.15@tcp231 [ 1667.044584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.160108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1667.222098] LNet: Added LNI 192.168.201.15@tcp231 [8/256/0/0] [ 1667.234369] LNet: 85866:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1667.570330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1667.597818] LNet: Removed LNI 192.168.201.15@tcp231 [ 1667.615833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1667.649179] LNet: Added LNI 192.168.201.15@tcp231 [8/256/0/0] [ 1668.871916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.891845] LNet: Removed LNI 192.168.201.15@tcp231 [ 1668.904842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.968381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1668.989341] LNet: Added LNI 192.168.201.15@tcp231 [8/256/0/60] [ 1668.994361] LNet: 86018:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1669.390831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1669.424573] LNet: Removed LNI 192.168.201.15@tcp231 [ 1669.445570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1669.490778] LNet: Added LNI 192.168.201.15@tcp231 [8/256/0/60] [ 1670.507609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.534203] LNet: Removed LNI 192.168.201.15@tcp231 [ 1670.552918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.623792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1670.654036] LNet: Added LNI 192.168.201.15@tcp231 [8/256/0/180] [ 1671.070578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1671.109575] LNet: Removed LNI 192.168.201.15@tcp231 [ 1671.130931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1671.172401] LNet: Added LNI 192.168.201.15@tcp231 [8/256/0/180] [ 1678.075245] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:25:25 (1788747925) [ 1678.199909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.254685] LNet: Removed LNI 192.168.201.15@tcp231 [ 1678.306974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.395359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1678.423168] LNet: Added LNI 192.168.201.15@tcp232 [8/256/0/180] [ 1679.113620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1679.148165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1679.176454] LNet: Removed LNI 192.168.201.15@tcp232 [ 1679.188506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1679.213350] LNet: Added LNI 192.168.201.15@tcp232 [8/256/0/180] [ 1679.867469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1679.896539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1679.921425] LNet: Removed LNI 192.168.201.15@tcp232 [ 1679.939364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1679.962508] LNet: Added LNI 192.168.201.15@tcp232 [8/256/0/180] [ 1680.674989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1680.720721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1680.753280] LNet: Removed LNI 192.168.201.15@tcp232 [ 1680.771827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1680.793156] LNet: Added LNI 192.168.201.15@tcp232 [8/256/0/180] [ 1681.572938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1681.611180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1681.640456] LNet: Removed LNI 192.168.201.15@tcp232 [ 1681.653802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1688.795535] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:25:35 (1788747935) [ 1690.308580] Key type .llcrypt unregistered [ 1690.310574] Key type ._llcrypt unregistered [ 1698.233237] Key type ._llcrypt registered [ 1698.234524] Key type .llcrypt registered [ 1698.618863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1698.638198] alg: No test for adler32 (adler32-zlib) [ 1699.650900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1699.674068] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1707.139798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 1716.871038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.916674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1716.936794] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1718.136134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_interrupt -r 2 [ 1718.174776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_interrupt -r 2 [ 1718.207215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_interrupt -r 2 [ 1718.238858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_interrupt -r 2 [ 1718.273840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_interrupt -r 2 [ 1718.312782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_interrupt -r 2 [ 1718.351544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_interrupt -r 2 [ 1718.390159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_interrupt -r 2 [ 1718.428183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.15@tcp 192.168.201.115@tcp [ 1718.442570] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1718.445836] LNet: 1 local NIs in recovery (showing 1): 192.168.201.15@tcp [ 1718.683654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1718.713634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1718.739436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1718.772839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1718.817925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1718.856295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1718.898946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1718.953871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1719.001133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1719.020679] LNet: 88670:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1719.052911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1719.102668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1719.165961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1719.203922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1719.246823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1719.301955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1719.866432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_dropped -r 2 [ 1719.920693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_dropped -r 2 [ 1719.956176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_dropped -r 2 [ 1720.032251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_dropped -r 2 [ 1720.110307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_dropped -r 2 [ 1720.170634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_dropped -r 2 [ 1720.240738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_dropped -r 2 [ 1720.333536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_dropped -r 2 [ 1720.402523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.15@tcp 192.168.201.115@tcp [ 1720.429443] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1720.783939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1720.817292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1720.855568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1720.886419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1720.943692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1720.999822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1721.059897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1721.105502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1721.162162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1721.182502] LNet: 88792:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1721.206702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1721.243723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1721.293685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1721.333389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1721.369577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1721.398172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1722.067831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_aborted -r 2 [ 1722.151730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_aborted -r 2 [ 1722.206886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_aborted -r 2 [ 1722.273030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_aborted -r 2 [ 1722.361258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_aborted -r 2 [ 1722.430981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_aborted -r 2 [ 1722.495299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_aborted -r 2 [ 1722.565361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_aborted -r 2 [ 1722.625781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.15@tcp 192.168.201.115@tcp [ 1722.661049] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1722.991298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1723.017842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1723.072733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1723.141248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1723.189617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1723.227522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1723.274684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1723.321705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1723.424185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1723.477055] LNet: 88915:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1723.501980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1723.576709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1723.618944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1723.665131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1723.698869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1723.742162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1724.425083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_no_route -r 2 [ 1724.470170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_no_route -r 2 [ 1724.559181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_no_route -r 2 [ 1724.626625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_no_route -r 2 [ 1724.657720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_no_route -r 2 [ 1724.688135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_no_route -r 2 [ 1724.750911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_no_route -r 2 [ 1724.806648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_no_route -r 2 [ 1724.866696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.15@tcp 192.168.201.115@tcp [ 1724.889469] LNet: There was an unexpected network error while writing to 192.168.201.115: rc = -22 [ 1725.163937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1725.189097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1725.232259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1725.275367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1725.318269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1725.346427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1725.381027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1725.411971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1725.452135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1725.492574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1725.525208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1725.564836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1725.603974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1725.639624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1725.674667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1726.190525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e local_timeout -r 2 [ 1726.226642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e local_timeout -r 2 [ 1726.267578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e local_timeout -r 2 [ 1726.312795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e local_timeout -r 2 [ 1726.370714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e local_timeout -r 2 [ 1726.419933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e local_timeout -r 2 [ 1726.474694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e local_timeout -r 2 [ 1726.524244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e local_timeout -r 2 [ 1726.555725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.15@tcp 192.168.201.115@tcp [ 1726.872199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1726.932847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1726.992882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1727.066951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1727.125278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1727.156990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1727.203818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1727.244437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1727.281036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1727.299520] LNet: 89159:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1727.305803] LNet: 89159:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1727.327126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1727.366516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1727.401735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1727.435602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1727.481379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1727.515344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1728.260585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp -m GET -e remote_dropped -r 2 [ 1728.287590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.115@tcp1 -m GET -e remote_dropped -r 2 [ 1728.306526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp -m GET -e remote_dropped -r 2 [ 1728.330996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp -d 192.168.201.15@tcp1 -m GET -e remote_dropped -r 2 [ 1728.362708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp -m GET -e remote_dropped -r 2 [ 1728.418475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.115@tcp1 -m GET -e remote_dropped -r 2 [ 1728.464443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp -m GET -e remote_dropped -r 2 [ 1728.531389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.15@tcp1 -d 192.168.201.15@tcp1 -m GET -e remote_dropped -r 2 [ 1728.578925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.15@tcp 192.168.201.115@tcp [ 1728.605416] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.115@tcp [ 1728.946462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1728.992202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1729.022721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1729.052785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1729.081377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1729.116071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1729.145418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1729.175934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1729.229603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1729.289970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1729.336636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1729.389917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1729.442938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1729.489139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1729.545552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1735.276425] LNet: 89356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.284593] LNetError: 89356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.351367] LNet: Removed LNI 192.168.201.15@tcp [ 1736.358521] LNetError: 89356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.376408] LNet: Removed LNI 192.168.201.15@tcp1 [ 1736.845241] Key type .llcrypt unregistered [ 1736.848588] Key type ._llcrypt unregistered [ 1755.595159] Key type ._llcrypt registered [ 1755.596981] Key type .llcrypt registered [ 1757.323360] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:26:43 (1788748003) [ 1757.686602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1757.703513] alg: No test for adler32 (adler32-zlib) [ 1758.633324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.678839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.696069] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1759.745250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1759.767960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.15@tcp [ 1759.785926] LNetError: 90335:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.15@tcp to recovery [ 1759.860764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.15@tcp [ 1759.963967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1767.499658] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:26:54 (1788748014) [ 1767.891352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.922519] LNet: 90924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.927952] LNetError: 90924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.951212] LNet: Removed LNI 192.168.201.15@tcp [ 1767.971334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.021680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1768.043385] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1768.769259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1768.797994] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 1769.782286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.15@tcp [ 1769.801952] LNetError: 91191:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.15@tcp to recovery [ 1770.017454] LNet: 1 local NIs in recovery (showing 1): 192.168.201.15@tcp [ 1770.026206] LNet: There was an unexpected network error while writing to 192.168.201.15: rc = -22 [ 1771.108633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.15@tcp [ 1778.825885] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:27:05 (1788748025) [ 1780.678482] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg115-server [ 1782.495538] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:27:09 (1788748029) [ 1784.991042] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg115-server [ 1787.629708] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:27:13 (1788748033) [ 1790.381892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg115-server [ 1793.453121] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:27:19 (1788748039) [ 1793.830355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.914919] LNet: 92814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.946655] LNetError: 92814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.973580] LNet: Removed LNI 192.168.201.15@tcp [ 1794.011139] LNet: Removed LNI 192.168.201.15@tcp1 [ 1794.107569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.238641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.284483] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1795.670654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1795.848282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1795.947853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1796.148837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1796.231285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1796.498317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1803.416561] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:27:30 (1788748050) [ 1803.545762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.611978] LNet: 93541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.622049] LNetError: 93541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.631907] LNetError: 93541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1803.691952] LNet: Removed LNI 192.168.201.15@tcp [ 1803.724283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.770402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1803.787930] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1811.231822] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:27:38 (1788748058) [ 1811.610910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.662466] LNet: 94251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.678331] LNetError: 94251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.698580] LNet: Removed LNI 192.168.201.15@tcp [ 1811.737503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.823420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1811.869859] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1820.945136] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:27:47 (1788748067) [ 1822.502731] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1824.501798] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:27:51 (1788748071) [ 1825.246513] LNet: 95286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.260932] LNetError: 95286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.269855] LNet: Removed LNI 192.168.201.15@tcp [ 1825.902993] Key type .llcrypt unregistered [ 1825.905889] Key type ._llcrypt unregistered [ 1832.847635] Key type ._llcrypt registered [ 1832.860901] Key type .llcrypt registered [ 1833.133708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1833.163653] alg: No test for adler32 (adler32-zlib) [ 1834.129772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1834.152556] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1841.486620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 1847.261415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.115@tcp [ 1847.688837] LNet: 95701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.707037] LNetError: 95701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.725572] LNet: Removed LNI 192.168.201.15@tcp [ 1848.273406] Key type .llcrypt unregistered [ 1848.279990] Key type ._llcrypt unregistered [ 1860.353395] Key type ._llcrypt registered [ 1860.355740] Key type .llcrypt registered [ 1861.661621] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1863.084550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1864.712455] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:28:31 (1788748111) [ 1866.360596] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1868.285560] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:28:35 (1788748115) [ 1870.334475] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg115-server [ 1873.194842] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:28:39 (1788748119) [ 1876.047379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg115-server [ 1878.393277] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:28:44 (1788748124) [ 1881.404704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1881.436029] alg: No test for adler32 (adler32-zlib) [ 1882.706631] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1882.957368] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1884.679284] Key type lgssc registered [ 1886.917076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1913.982808] Key type lgssc unregistered [ 1914.366315] LNet: 98874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.385092] LNetError: 98874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.428821] LNet: Removed LNI 192.168.201.15@tcp [ 1915.320121] Key type .llcrypt unregistered [ 1915.322823] Key type ._llcrypt unregistered [ 1929.738204] Key type ._llcrypt registered [ 1929.740609] Key type .llcrypt registered [ 1930.751769] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:29:38 (1788748178) [ 1931.228255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1931.242613] alg: No test for adler32 (adler32-zlib) [ 1932.109514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.162571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1932.203559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1932.286690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1932.358078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1937.988201] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:29:45 (1788748185) [ 1938.538900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1938.596103] LNet: 100286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.631320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.702223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1938.735577] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1939.593490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1939.613858] LNet: 100430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.630193] LNetError: 100430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.646783] LNet: Removed LNI 192.168.201.15@tcp [ 1939.703405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.804815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1939.849548] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1940.409086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1940.429832] LNetError: 100458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.444357] LNet: Removed LNI 192.168.201.15@tcp [ 1940.473902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1940.530993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1940.556904] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1948.058813] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:29:54 (1788748194) [ 1948.226219] LNet: 101039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.239653] LNet: 101039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1948.259083] LNetError: 101039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.276109] LNet: Removed LNI 192.168.201.15@tcp [ 1949.066729] Key type .llcrypt unregistered [ 1949.069052] Key type ._llcrypt unregistered [ 1956.925601] Key type ._llcrypt registered [ 1956.927951] Key type .llcrypt registered [ 1957.202626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1957.250133] alg: No test for adler32 (adler32-zlib) [ 1958.238671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.334214] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 1958.340993] LNet: *** cfs_fail_loc=e003, val=100*** [ 1959.254052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 1959.270598] LNet: *** cfs_fail_loc=e003, val=100*** [ 1959.272867] LNet: Skipped 2 previous similar messages [ 1959.278698] Ping reply received with mapping: [ 1959.281832] found metadata reply NI[192.168.201.15@tcp] [ 1959.286421] 0f c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1959.289777] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1959.292896] 5a 5a 5a 5a 5a 5a 5a 5a [ 1959.528264] LNet: 101443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.534299] LNetError: 101443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.554222] LNet: Removed LNI 192.168.201.15@tcp [ 1961.481410] Key type .llcrypt unregistered [ 1961.485441] Key type ._llcrypt unregistered [ 1977.452135] Key type ._llcrypt registered [ 1977.461193] Key type .llcrypt registered [ 1979.128789] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:30:25 (1788748225) [ 1979.595428] Key type .llcrypt unregistered [ 1979.598510] Key type ._llcrypt unregistered [ 1988.290300] Key type ._llcrypt registered [ 1988.291753] Key type .llcrypt registered [ 1988.785043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1988.826255] alg: No test for adler32 (adler32-zlib) [ 1990.432396] LNetError: 102481:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1991.008732] Key type .llcrypt unregistered [ 1991.010766] Key type ._llcrypt unregistered [ 1998.853334] Key type ._llcrypt registered [ 1998.861778] Key type .llcrypt registered [ 2000.996380] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:30:47 (1788748247) [ 2001.980343] Key type .llcrypt unregistered [ 2001.985255] Key type ._llcrypt unregistered [ 2011.449107] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2011.463128] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2014.265364] Key type ._llcrypt registered [ 2014.267701] Key type .llcrypt registered [ 2014.451439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2014.490338] alg: No test for adler32 (adler32-zlib) [ 2015.398107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2015.445854] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2027.162414] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:31:14 (1788748274) [ 2027.313941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2027.338211] LNet: 104256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.345343] LNetError: 104256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2027.377364] LNet: Removed LNI 10.1.2.3@tcp [ 2027.407639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.049420] Key type .llcrypt unregistered [ 2028.051981] Key type ._llcrypt unregistered [ 2035.707805] Key type ._llcrypt registered [ 2035.710747] Key type .llcrypt registered [ 2035.911439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2035.940764] alg: No test for adler32 (adler32-zlib) [ 2036.984831] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2037.071684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2037.317298] LNet: 104541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.333753] LNetError: 104541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.353428] LNet: Removed LNI 192.168.201.15@tcp [ 2038.108746] Key type .llcrypt unregistered [ 2038.111914] Key type ._llcrypt unregistered [ 2050.841283] Key type ._llcrypt registered [ 2050.843076] Key type .llcrypt registered [ 2052.401285] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:31:39 (1788748299) [ 2053.178305] Key type .llcrypt unregistered [ 2053.182538] Key type ._llcrypt unregistered [ 2061.417278] Key type ._llcrypt registered [ 2061.423532] Key type .llcrypt registered [ 2061.857662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2061.875379] alg: No test for adler32 (adler32-zlib) [ 2071.833804] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:31:58 (1788748318) [ 2072.067167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2072.127512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.335637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2072.438378] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2073.190241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2073.233409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2073.260560] LNet: Removed LNI 192.168.201.15@tcp [ 2073.551851] LNet: 106281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.017568] Key type .llcrypt unregistered [ 2074.020965] Key type ._llcrypt unregistered [ 2084.966735] Key type ._llcrypt registered [ 2084.968950] Key type .llcrypt registered [ 2085.958619] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:32:13 (1788748333) [ 2086.346841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2086.363194] alg: No test for adler32 (adler32-zlib) [ 2087.261434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.299610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.320369] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2088.176329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2088.206743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.15@tcp [ 2088.226799] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2088.229269] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2088.231311] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2088, 0, 0 [ 2088.233833] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2088.235444] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2088.237607] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2088.239520] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2088, 0, 0 [ 2088.246563] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2088.249558] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2088.254504] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2088.256418] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2088, 0, 2138 [ 2088.258835] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2088.260763] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2088.262685] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2088.265807] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2088, 0, 2138 [ 2088.270654] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2094.443475] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:32:21 (1788748341) [ 2094.724813] LNet: 107817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.731190] LNetError: 107817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.749670] LNet: Removed LNI 192.168.201.15@tcp [ 2095.238883] Key type .llcrypt unregistered [ 2095.246469] Key type ._llcrypt unregistered [ 2102.418113] Key type ._llcrypt registered [ 2102.421281] Key type .llcrypt registered [ 2102.745203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2102.777413] alg: No test for adler32 (adler32-zlib) [ 2103.803531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2103.861142] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2111.982861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 2121.472522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.511689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2121.545693] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 2122.582327] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2122.769279] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2122.773251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2122.784201] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2125.109528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2125.127577] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2136.041584] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:33:02 (1788748382) [ 2136.283190] LNet: 109316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.291899] LNetError: 109316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.303651] LNet: Removed LNI 192.168.201.15@tcp [ 2137.312595] LNet: Removed LNI 192.168.201.15@tcp1 [ 2137.322939] LNetError: 109316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2137.341744] LNetError: 109316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2137.372172] LNet: Removed LNI 10.1.2.3@tcp99 [ 2138.183493] Key type .llcrypt unregistered [ 2138.186189] Key type ._llcrypt unregistered [ 2144.879862] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2144.895342] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2147.473611] Key type ._llcrypt registered [ 2147.480371] Key type .llcrypt registered [ 2147.602206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2147.615706] alg: No test for adler32 (adler32-zlib) [ 2148.470886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.510249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2148.529575] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2149.280732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2149.320918] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2151.488361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2151.556569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2151.627868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2151.685367] LNetError: 110063:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2151.750715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2151.835781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2152.397765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2152.467244] LNet: 110083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.478652] LNetError: 110083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2152.483418] LNet: Removed LNI 10.1.2.3@tcp [ 2153.511807] LNet: Removed LNI 192.168.201.15@tcp [ 2154.436398] Key type .llcrypt unregistered [ 2154.447741] Key type ._llcrypt unregistered [ 2166.373375] Key type ._llcrypt registered [ 2166.375539] Key type .llcrypt registered [ 2167.581372] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:33:34 (1788748414) [ 2167.897145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2167.904168] alg: No test for adler32 (adler32-zlib) [ 2168.860661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.967657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2168.994274] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2169.608622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2169.674995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg115-client@tcp [ 2174.923848] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:33:42 (1788748422) [ 2175.042089] LNet: 111610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2175.046425] LNetError: 111610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.066680] LNet: Removed LNI 192.168.201.15@tcp [ 2175.627393] Key type .llcrypt unregistered [ 2175.628896] Key type ._llcrypt unregistered [ 2182.413197] Key type ._llcrypt registered [ 2182.417159] Key type .llcrypt registered [ 2182.623863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.646066] alg: No test for adler32 (adler32-zlib) [ 2183.647226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2183.670126] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2183.710553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2190.471899] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:33:57 (1788748437) [ 2192.312724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2193.875560] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:34:01 (1788748441) [ 2195.737832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2197.318641] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:34:04 (1788748444) [ 2197.415691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2197.441702] LNet: 113031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2197.445725] LNetError: 113031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2197.455921] LNet: Removed LNI 192.168.201.15@tcp [ 2197.471406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.517764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2197.534899] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2198.211709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2198.300575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2204.726235] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:34:11 (1788748451) [ 2204.826747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2204.873071] LNet: 113739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.884348] LNetError: 113739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.900849] LNet: Removed LNI 192.168.201.15@tcp [ 2204.928487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.010765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2205.038933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2205.565380] Key type .llcrypt unregistered [ 2205.569726] Key type ._llcrypt unregistered [ 2216.707886] Key type ._llcrypt registered [ 2216.709759] Key type .llcrypt registered [ 2217.914871] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:34:25 (1788748465) [ 2218.490937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2218.512376] alg: No test for adler32 (adler32-zlib) [ 2219.487177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.578176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2219.616707] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2220.732900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2220.814174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2227.380983] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:34:34 (1788748474) [ 2227.521646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.557208] LNet: 115295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.567372] LNetError: 115295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.622453] LNet: Removed LNI 192.168.201.15@tcp [ 2227.646270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.714571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2227.768937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2234.779739] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:34:41 (1788748481) [ 2234.853646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.879885] LNet: 115872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.896123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.944701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2235.178509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2235.207703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2235.258871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2235.594722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2235.645577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2235.712220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2235.753510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.838968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2236.347790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2236.382695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2236.441858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2236.917856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2236.973974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2244.483915] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:34:51 (1788748491) [ 2244.982363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.015353] LNet: 116514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.022115] LNet: 116514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2245.057898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.151758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2245.193062] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2246.259539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2246.312253] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2247.536886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2247.618824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2247.930428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.980504] LNet: 116793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2247.990347] LNetError: 116793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.001612] LNet: Removed LNI 192.168.201.15@tcp [ 2248.027860] LNet: Removed LNI 192.168.201.15@tcp2 [ 2248.079189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.155432] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2248.166293] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2248.174074] LNet: 116799:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2249.080206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2249.426066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2249.462540] LNetError: 116837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.468701] LNetError: 116837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2249.475365] LNet: Removed LNI 192.168.201.15@tcp [ 2249.487812] LNet: Removed LNI 192.168.201.15@tcp2 [ 2249.507861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2249.563435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2249.606510] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2250.471408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2250.496788] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2251.291320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2251.368585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2251.469457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2251.574661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.585056] LNetError: 117121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.591175] LNetError: 117121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2251.595207] LNet: Removed LNI 192.168.201.15@tcp [ 2252.661387] LNet: Removed LNI 192.168.201.15@tcp2 [ 2252.714532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.798575] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2252.806843] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2252.816506] LNet: 117127:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2253.368898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2253.521355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2253.535509] LNet: 117164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.539567] LNet: 117164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2253.543460] LNet: Removed LNI 192.168.201.15@tcp [ 2254.572065] LNet: Removed LNI 192.168.201.15@tcp2 [ 2254.637281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.703955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2254.752724] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2255.608938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2255.634348] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2256.553515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2256.609495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2256.785319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.807802] LNetError: 117444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.811799] LNetError: 117444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2256.816489] LNet: Removed LNI 192.168.201.15@tcp [ 2256.824782] LNet: Removed LNI 192.168.201.15@tcp2 [ 2256.837609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.877280] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2256.881526] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2256.885752] LNet: 117450:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2257.260387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2257.528741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.591398] LNet: Removed LNI 192.168.201.15@tcp [ 2258.605564] LNet: Removed LNI 192.168.201.15@tcp2 [ 2258.656886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.765481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2258.808297] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2259.500336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2259.517322] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2260.191638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2260.255120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2260.353525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2260.515960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.599192] LNet: Removed LNI 192.168.201.15@tcp [ 2262.630300] LNet: Removed LNI 192.168.201.15@tcp2 [ 2262.681848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.770170] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2262.778622] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2262.781954] LNet: 117778:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2263.271381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2263.467589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.487737] LNet: 117815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.491939] LNet: 117815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2263.497816] LNet: Removed LNI 192.168.201.15@tcp [ 2264.551074] LNet: Removed LNI 192.168.201.15@tcp2 [ 2264.609076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.673680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2264.714802] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2265.444859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2265.472460] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2266.225705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2266.282743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2266.354963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2266.713325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2266.724609] LNetError: 118111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.732679] LNetError: 118111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2266.743615] LNet: Removed LNI 192.168.201.15@tcp [ 2266.765567] LNet: Removed LNI 192.168.201.15@tcp2 [ 2266.810624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.901467] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2266.904703] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2266.909498] LNet: 118117:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2267.521633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2267.878229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.959174] LNet: Removed LNI 192.168.201.15@tcp [ 2268.975120] LNet: Removed LNI 192.168.201.15@tcp2 [ 2269.021407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.103300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2269.143784] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2269.715660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2269.737074] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2270.380308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2270.416580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2270.439373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2270.482269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2270.749709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.775267] LNet: Removed LNI 192.168.201.15@tcp [ 2272.805538] LNet: Removed LNI 192.168.201.15@tcp2 [ 2272.882415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.948232] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2272.953774] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2273.510418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2280.021095] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2281.501601] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:35:28 (1788748528) [ 2281.609840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.643102] LNet: 119229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.653395] LNet: 119229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2281.662956] LNet: Removed LNI 192.168.201.15@tcp [ 2282.729360] LNet: Removed LNI 192.168.201.15@tcp2 [ 2282.768243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.108412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2283.132522] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2283.936505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2283.964341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2284.650772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.667765] LNetError: 119510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2284.673513] LNetError: 119510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2284.678648] LNet: Removed LNI 10.1.2.3@tcp [ 2284.687864] LNet: Removed LNI 192.168.201.15@tcp [ 2284.703860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.774495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2284.801383] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2284.806527] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2285.252091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2285.273647] LNet: Removed LNI 192.168.201.15@tcp [ 2285.285855] LNet: Removed LNI 10.1.2.3@tcp [ 2290.739264] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:35:37 (1788748537) [ 2291.053654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2291.147945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2291.249089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2291.278249] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2292.028385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2292.057419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2292.105032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2292.164610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2292.239177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2292.288826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2292.335156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2292.428218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2292.487802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2292.543304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2292.602973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2292.660636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2292.858422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2299.171590] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 22:35:46 (1788748546) [ 2299.343464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2300.462461] LNet: Removed LNI 192.168.201.15@tcp [ 2300.515369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.599984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.653817] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2301.635594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2301.676879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2301.732282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2301.774196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2301.824760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2301.849986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2301.897888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2301.938467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2301.969489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2301.993635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.025392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.075878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.195949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2302.276517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.349185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.400572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.434897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.478062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.509784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.542893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.602699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.645420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2302.678405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.15@tcp [ 2309.084403] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:35:56 (1788748556) [ 2309.218715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2310.314649] LNet: Removed LNI 192.168.201.15@tcp [ 2310.381454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.493695] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2310.624249] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2310.634358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2310.646494] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2313.035866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2313.076342] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2313.822574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2313.842308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2315.726015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2315.755439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2315.854173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.15@tcp -m PUT -e local_timeout -r 1 [ 2315.953519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2316.062320] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2316.064641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2316.304926] LNetError: 121674:0:(lib-move.c:4794:lnet_parse()) 192.168.201.15@tcp, src 192.168.201.15@tcp: Dropping PUT (error -108 looking up sender) [ 2317.343393] LNet: 122132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.350106] LNet: 122132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2317.357673] LNetError: 122132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2317.363462] LNetError: 122132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2317.368272] LNet: Removed LNI 10.1.2.3@tcp [ 2318.437519] LNet: Removed LNI 192.168.201.15@tcp [ 2318.848738] Key type .llcrypt unregistered [ 2318.851508] Key type ._llcrypt unregistered [ 2330.310296] Key type ._llcrypt registered [ 2330.312627] Key type .llcrypt registered [ 2331.799413] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:36:18 (1788748578) [ 2332.169735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2332.180183] alg: No test for adler32 (adler32-zlib) [ 2333.200291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.782740] LNet: 122981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.788405] Key type .llcrypt unregistered [ 2334.790223] Key type ._llcrypt unregistered [ 2341.678528] Key type ._llcrypt registered [ 2341.681902] Key type .llcrypt registered [ 2341.855656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2341.869284] alg: No test for adler32 (adler32-zlib) [ 2342.788870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2342.819952] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2349.608287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 2349.773210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2350.331855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.115@tcp [ 2350.509893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2350.525899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2350.576200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2350.608882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2350.654390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2350.716319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2350.803173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2350.844320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2350.888817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2350.914597] LNet: 123430:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2350.949110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2351.004395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2351.008774] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.115@tcp [ 2351.054039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2351.081973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2351.107390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2351.137676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2351.651967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2351.976445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2352.013753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2352.043268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2352.081689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2352.107375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2352.134684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2352.167860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2352.213220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2352.245656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2352.272209] LNet: 123549:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2352.289938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2352.325788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2352.367357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2352.409572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2352.452391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2352.487818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2353.043906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.115@tcp [ 2353.278859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2353.307513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2353.340769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2353.382694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2353.409556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2353.440480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2353.469960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2353.505822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2353.541714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2353.561841] LNet: 123669:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.607723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2353.664695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2353.738292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2353.810770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2353.846418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2353.877194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2354.496335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2354.671628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2354.700643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2354.729956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2354.765872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2354.807732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2354.845879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2354.883499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2354.920657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2354.965898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2355.003589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2355.056627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2355.102793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2355.176550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2355.240501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2355.293393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2355.612491] LNet: 123834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.618842] LNetError: 123834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.633186] LNet: Removed LNI 192.168.201.15@tcp [ 2356.078750] Key type .llcrypt unregistered [ 2356.085431] Key type ._llcrypt unregistered [ 2375.197741] Key type ._llcrypt registered [ 2375.199908] Key type .llcrypt registered [ 2376.388849] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:37:03 (1788748623) [ 2376.820302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2376.827587] alg: No test for adler32 (adler32-zlib) [ 2377.787447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.232063] LNet: 124719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.944797] Key type .llcrypt unregistered [ 2378.947877] Key type ._llcrypt unregistered [ 2386.907301] Key type ._llcrypt registered [ 2386.910516] Key type .llcrypt registered [ 2387.177955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2387.193266] alg: No test for adler32 (adler32-zlib) [ 2388.177753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2388.200575] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2397.060384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.115@tcp [ 2409.431476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.509350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2409.533542] LNet: Added LNI 192.168.201.15@tcp1 [8/256/0/180] [ 2410.983381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2411.821457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.115@tcp [ 2412.110323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2412.142883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2412.169938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2412.202226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2412.235878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2412.278792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2412.325663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2412.382992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2412.457784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2412.491876] LNet: 125373:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2412.534216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2412.607602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2412.671913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2412.748933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2412.768140] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.115@tcp [ 2412.832946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2412.916793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2413.891850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2414.133473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2414.179859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2414.257636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2414.292954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2414.325830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2414.363285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2414.399605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2414.450932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2414.522994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2414.564816] LNet: 125492:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2414.592705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2414.641671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2414.677961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2414.707620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2414.735486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2414.763991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2415.525651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.115@tcp [ 2415.912538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2415.953786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2415.992663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2416.043223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2416.099949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2416.136049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2416.166987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2416.206508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2416.242844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2416.275256] LNet: 125611:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2416.294716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2416.337139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2416.376852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2416.422519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2416.459489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2416.505301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2417.499695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2417.797695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2417.841313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2417.915906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2417.953546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2418.006983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2418.054778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2418.086978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2418.124396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2418.162222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2418.204420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2418.264836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2418.316106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2418.382835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2418.455325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2418.519521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2424.402560] LNet: 125812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.407786] LNetError: 125812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.412540] LNet: Removed LNI 192.168.201.15@tcp [ 2424.421885] LNet: Removed LNI 192.168.201.15@tcp1 [ 2424.872729] Key type .llcrypt unregistered [ 2424.874358] Key type ._llcrypt unregistered [ 2442.620710] Key type ._llcrypt registered [ 2442.630912] Key type .llcrypt registered [ 2443.945742] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 22:38:10 (1788748690) [ 2444.921908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2444.955269] alg: No test for adler32 (adler32-zlib) [ 2445.990230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.113780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2446.175958] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2447.442971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2447.469542] LNet: Added LNI 192.168.201.15@tcp2 [8/256/0/180] [ 2448.693409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.15@tcp [ 2448.840657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2449.887610] LNet: Removed LNI 192.168.201.15@tcp [ 2457.191326] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 22:38:24 (1788748704) [ 2458.872504] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2460.580294] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:38:27 (1788748707) [ 2468.780161] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:38:35 (1788748715) [ 2476.966682] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:38:43 (1788748723) [ 2486.371088] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:38:52 (1788748732) [ 2494.857616] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:39:02 (1788748742) [ 2502.631087] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:39:09 (1788748749) [ 2511.765336] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:39:18 (1788748758) [ 2519.974689] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:39:26 (1788748766) [ 2528.042071] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:39:35 (1788748775) [ 2535.221681] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:39:42 (1788748782) [ 2542.259546] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:39:49 (1788748789) [ 2549.476120] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:39:56 (1788748796) [ 2555.562972] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:40:02 (1788748802) [ 2563.482781] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:40:10 (1788748810) [ 2569.739349] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:40:16 (1788748816) [ 2575.806349] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:40:23 (1788748823) [ 2582.282112] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:40:29 (1788748829) [ 2589.331512] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:40:36 (1788748836) [ 2589.383905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2589.401090] LNet: 137506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.407578] LNetError: 137506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.423885] LNet: Removed LNI 192.168.201.15@tcp2 [ 2589.462524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2589.589743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2589.605434] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2590.617225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2590.644449] LNet: 137656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2590.655115] LNetError: 137656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2590.671705] LNet: Removed LNI 192.168.201.15@tcp [ 2590.699338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2590.744359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2590.787055] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2596.890759] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:40:44 (1788748844) [ 2603.991442] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:40:51 (1788748851) [ 2610.517775] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:40:57 (1788748857) [ 2617.509595] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:41:04 (1788748864) [ 2623.581502] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:41:10 (1788748870) [ 2629.706151] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:41:16 (1788748876) [ 2635.611417] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:41:23 (1788748883) [ 2641.381810] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:41:28 (1788748888) [ 2647.194933] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:41:34 (1788748894) [ 2652.647182] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:41:40 (1788748900) [ 2658.509092] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:41:45 (1788748905) [ 2663.853269] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:41:51 (1788748911) [ 2670.556588] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2417 sec ========== 22:41:57 (1788748917) [ 2670.835607] LNet: 145117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2670.843891] LNetError: 145117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2670.858237] LNet: Removed LNI 192.168.201.15@tcp [ 2671.361765] Key type .llcrypt unregistered [ 2671.364268] Key type ._llcrypt unregistered [ 2681.809581] Key type ._llcrypt registered [ 2681.817044] Key type .llcrypt registered [ 2682.099813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2682.110566] alg: No test for adler32 (adler32-zlib) [ 2683.138960] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 2683.338683] LNet: Added LNI 192.168.201.15@tcp [8/256/0/180] [ 2684.976310] Key type lgssc registered [ 2686.402127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2743.138577] Lustre: Mounted lustre-client - version 2.17.58_39_g34d8757 [ 2747.786882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2754.078370] Lustre: Unmounted lustre-client [ 2775.841446] Key type lgssc unregistered [ 2776.079942] LNet: 148077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2776.092139] LNetError: 148077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2776.108882] LNet: Removed LNI 192.168.201.15@tcp [ 2776.765411] Key type .llcrypt unregistered [ 2776.768249] Key type ._llcrypt unregistered