[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 556814408 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003349] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.008753] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011019] pid_max: default: 32768 minimum: 301 [ 0.012168] LSM: Security Framework initializing [ 0.013072] Yama: becoming mindful. [ 0.014050] SELinux: Initializing. [ 0.015097] *** VALIDATE selinux *** [ 0.023811] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028637] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029210] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032188] *** VALIDATE tmpfs *** [ 0.033624] *** VALIDATE proc *** [ 0.035457] *** VALIDATE cgroup *** [ 0.036021] *** VALIDATE cgroup2 *** [ 0.037336] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041027] Spectre V2 : User space: Vulnerable [ 0.042014] Speculative Store Bypass: Vulnerable [ 0.045488] debug: unmapping init [mem 0xffffffffb9859000-0xffffffffb9860fff] [ 0.048306] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049881] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050037] ... version: 2 [ 0.051023] ... bit width: 48 [ 0.052033] ... generic registers: 4 [ 0.053021] ... value mask: 0000ffffffffffff [ 0.054021] ... max period: 00007fffffffffff [ 0.055022] ... fixed-purpose events: 3 [ 0.056013] ... event mask: 000000070000000f [ 0.057418] rcu: Hierarchical SRCU implementation. [ 0.059859] smp: Bringing up secondary CPUs ... [ 0.060731] x86: Booting SMP configuration: [ 0.061040] .... node #0, CPUs: #1 #2 #3 [ 0.065599] smp: Brought up 1 node, 4 CPUs [ 0.067020] smpboot: Max logical packages: 1 [ 0.068021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.201570] node 0 deferred pages initialised in 130ms [ 0.203193] devtmpfs: initialized [ 0.205288] x86/mm: Memory block size: 128MB [ 0.209674] gcov: version magic: 0x41383552 [ 0.212348] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.217156] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.220363] pinctrl core: initialized pinctrl subsystem [ 0.222200] [ 0.223008] ************************************************************* [ 0.226016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228011] ** ** [ 0.231014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.233013] ** ** [ 0.236018] ** This means that this kernel is built to expose internal ** [ 0.239017] ** IOMMU data structures, which may compromise security on ** [ 0.241011] ** your system. ** [ 0.243013] ** ** [ 0.246010] ** If you see this message and you are not debugging the ** [ 0.248012] ** kernel, report this immediately to your vendor! ** [ 0.250007] ** ** [ 0.252010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253007] ************************************************************* [ 0.256927] NET: Registered protocol family 16 [ 0.258533] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.263066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.266010] cpuidle: using governor menu [ 0.267824] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.269549] PCI: Using configuration type 1 for base access [ 0.271109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.278195] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.279015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.282111] cryptd: max_cpu_qlen set to 1000 [ 0.285265] ACPI: Added _OSI(Module Device) [ 0.287011] ACPI: Added _OSI(Processor Device) [ 0.289014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.291013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.296841] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.304080] ACPI: Interpreter enabled [ 0.305127] ACPI: PM: (supports S0 S3 S4 S5) [ 0.307011] ACPI: Using IOAPIC for interrupt routing [ 0.308176] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.312565] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.325823] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.328043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.331019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.335088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.340684] acpiphp: Slot [2] registered [ 0.342158] acpiphp: Slot [5] registered [ 0.344137] acpiphp: Slot [6] registered [ 0.345237] acpiphp: Slot [3] registered [ 0.347099] acpiphp: Slot [4] registered [ 0.349098] acpiphp: Slot [7] registered [ 0.350139] acpiphp: Slot [8] registered [ 0.352094] acpiphp: Slot [9] registered [ 0.353129] acpiphp: Slot [10] registered [ 0.355106] acpiphp: Slot [11] registered [ 0.356108] acpiphp: Slot [12] registered [ 0.358094] acpiphp: Slot [13] registered [ 0.359194] acpiphp: Slot [14] registered [ 0.361109] acpiphp: Slot [15] registered [ 0.363185] acpiphp: Slot [16] registered [ 0.364235] acpiphp: Slot [17] registered [ 0.366121] acpiphp: Slot [18] registered [ 0.368131] acpiphp: Slot [19] registered [ 0.370120] acpiphp: Slot [20] registered [ 0.371171] acpiphp: Slot [21] registered [ 0.373127] acpiphp: Slot [22] registered [ 0.374105] acpiphp: Slot [23] registered [ 0.376116] acpiphp: Slot [24] registered [ 0.378178] acpiphp: Slot [25] registered [ 0.379154] acpiphp: Slot [26] registered [ 0.381094] acpiphp: Slot [27] registered [ 0.382096] acpiphp: Slot [28] registered [ 0.384114] acpiphp: Slot [29] registered [ 0.386093] acpiphp: Slot [30] registered [ 0.387095] acpiphp: Slot [31] registered [ 0.388062] PCI host bridge to bus 0000:00 [ 0.390020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.393022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.395021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.398025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.401026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.404022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.405307] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.408131] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.412480] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.421017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.426000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.427080] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.430022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.433106] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.436375] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.438994] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.442053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.446012] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.450014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.460807] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.465719] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.473077] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.478017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.484016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.495018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.503950] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.515020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.524018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.550019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.566116] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.568387] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.571513] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.573410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.576225] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.583014] iommu: Default domain type: Passthrough [ 0.585551] SCSI subsystem initialized [ 0.587168] ACPI: bus type USB registered [ 0.589158] usbcore: registered new interface driver usbfs [ 0.592129] usbcore: registered new interface driver hub [ 0.595149] usbcore: registered new device driver usb [ 0.597188] pps_core: LinuxPPS API ver. 1 registered [ 0.599012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.602078] PTP clock support registered [ 0.604144] EDAC MC: Ver: 3.0.0 [ 0.606148] PCI: Using ACPI for IRQ routing [ 0.609327] NetLabel: Initializing [ 0.611013] NetLabel: domain hash size = 128 [ 0.613010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.615095] NetLabel: unlabeled traffic allowed by default [ 0.617135] vgaarb: loaded [ 0.618272] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.620014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.627000] clocksource: Switched to clocksource kvm-clock [ 0.810904] VFS: Disk quotas dquot_6.6.0 [ 0.812200] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.814142] *** VALIDATE ramfs *** [ 0.815226] *** VALIDATE hugetlbfs *** [ 0.816725] pnp: PnP ACPI init [ 0.818884] pnp: PnP ACPI: found 6 devices [ 0.835205] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.838098] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.839973] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.841736] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.843780] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.845779] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.848330] NET: Registered protocol family 2 [ 0.852719] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.859117] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.862268] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.867048] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.870913] TCP: Hash tables configured (established 65536 bind 65536) [ 0.874513] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.877418] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880238] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.883886] NET: Registered protocol family 1 [ 0.886691] RPC: Registered named UNIX socket transport module. [ 0.889238] RPC: Registered udp transport module. [ 0.890937] RPC: Registered tcp transport module. [ 0.894161] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.897523] NET: Registered protocol family 44 [ 0.899815] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.902924] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.904714] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.906815] PCI: CLS 0 bytes, default 64 [ 0.908206] Unpacking initramfs... [ 2.926693] debug: unmapping init [mem 0xffff8e81fcc64000-0xffff8e81fffcffff] [ 2.931256] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.933738] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.937052] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.560645] Initialise system trusted keyrings [ 3.562565] Key type blacklist registered [ 3.564664] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.574805] zbud: loaded [ 3.577924] *** VALIDATE nfs *** [ 3.579037] *** VALIDATE nfs4 *** [ 3.580632] pstore: using deflate compression [ 3.584650] Platform Keyring initialized [ 3.706973] NET: Registered protocol family 38 [ 3.708482] Key type asymmetric registered [ 3.709975] Asymmetric key parser 'x509' registered [ 3.713408] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.721181] io scheduler mq-deadline registered [ 3.722784] io scheduler kyber registered [ 3.724091] io scheduler bfq registered [ 3.726376] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.729991] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.733680] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.738537] ACPI: Power Button [PWRF] [ 3.744775] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.752208] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.762329] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.792250] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.821966] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.826856] Non-volatile memory driver v1.3 [ 3.828641] Linux agpgart interface v0.103 [ 3.863404] virtio_blk virtio1: [vda] 146216 512-byte logical blocks (74.9 MB/71.4 MiB) [ 3.866341] vda: detected capacity change from 0 to 74862592 [ 3.882243] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.885799] vdb: detected capacity change from 0 to 1073741824 [ 3.895695] libphy: Fixed MDIO Bus: probed [ 3.904936] usbcore: registered new interface driver usbserial_generic [ 3.909593] usbserial: USB Serial support registered for generic [ 3.912443] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.918839] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.921447] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.923846] mousedev: PS/2 mouse device common for all mice [ 3.927261] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.927260] rtc_cmos 00:05: RTC can wake from S4 [ 3.938576] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.941852] rtc_cmos 00:05: registered as rtc0 [ 3.948879] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.949539] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.949583] intel_pstate: CPU model not supported [ 3.956239] hid: raw HID events driver (C) Jiri Kosina [ 3.963985] usbcore: registered new interface driver usbhid [ 3.965615] usbhid: USB HID core driver [ 3.967343] drop_monitor: Initializing network drop monitor service [ 3.969859] Initializing XFRM netlink socket [ 3.972057] NET: Registered protocol family 10 [ 3.975164] Segment Routing with IPv6 [ 3.976711] NET: Registered protocol family 17 [ 3.978881] mpls_gso: MPLS GSO support [ 3.985409] RAS: Correctable Errors collector initialized. [ 3.987587] AVX version of gcm_enc/dec engaged. [ 3.989246] AES CTR mode by8 optimization enabled [ 4.095221] sched_clock: Marking stable (4095198237, 0)->(5154246683, -1059048446) [ 4.100369] registered taskstats version 1 [ 4.102524] Loading compiled-in X.509 certificates [ 4.105569] zswap: loaded using pool lzo/zbud [ 4.136781] Key type big_key registered [ 4.155758] Key type encrypted registered [ 4.158599] ima: No TPM chip found, activating TPM-bypass! [ 4.160937] ima: Allocated hash algorithm: sha1 [ 4.163829] ima: No architecture policies found [ 4.165933] evm: Initialising EVM extended attributes: [ 4.167853] evm: security.selinux [ 4.169257] evm: security.ima [ 4.170620] evm: security.capability [ 4.172317] evm: HMAC attrs: 0x1 [ 4.175163] rtc_cmos 00:05: setting system clock to 2026-07-28 06:18:30 UTC (1785219510) [ 4.183566] debug: unmapping init [mem 0xffffffffba803000-0xffffffffba9fffff] [ 4.187393] debug: unmapping init [mem 0xffffffffb9582000-0xffffffffb9858fff] [ 4.199553] Write protecting the kernel read-only data: 28672k [ 4.204571] debug: unmapping init [mem 0xffffffffb7c03000-0xffffffffb7dfffff] [ 4.206912] debug: unmapping init [mem 0xffffffffb8514000-0xffffffffb85fffff] [ 4.244327] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.250255] systemd[1]: Detected virtualization kvm. [ 4.252386] systemd[1]: Detected architecture x86-64. [ 4.254624] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.282968] systemd[1]: No hostname configured. [ 4.284145] systemd[1]: Set hostname to . [ 4.285483] random: systemd: uninitialized urandom read (16 bytes read) [ 4.287053] systemd[1]: Initializing machine ID from random generator. [ 4.386853] random: ln: uninitialized urandom read (6 bytes read) [ 4.577847] random: systemd: uninitialized urandom read (16 bytes read) [ 4.580227] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.594241] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.605538] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.573994] device-mapper: uevent: version 1.0.3 [ 5.577801] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.923077] random: fast init done [ 8.000250] virtio_net virtio0 ens2: renamed from eth0 [ 8.406059] scsi host0: ata_piix [ 8.443719] scsi host1: ata_piix [ 8.445194] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.467340] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 13.690043] random: crng init done [ 13.691439] random: 7 urandom warning(s) missed due to ratelimiting [ 17.180899] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.136413] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.878391] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.843626] SELinux: Disabled at runtime. [ 23.968946] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.985247] systemd[1]: Detected virtualization kvm. [ 23.988858] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.872357] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.883616] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.921260] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.936040] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.956029] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.995975] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.059215] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ 26.652266] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 27.926829] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.646065] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.687579] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.265613] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.452379] EDAC sbridge: Ver: 1.1.2 [ 33.513943] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit)[ 34.716474] NFS: Registering the id_resolver key type [ 34.722765] Key type id_resolver registered [ 34.725756] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg245-client login: [ 98.384512] libcfs: loading out-of-tree module taints kernel. [ 98.528436] Key type ._llcrypt registered [ 98.555404] Key type .llcrypt registered [ 99.480776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 99.498750] alg: No test for adler32 (adler32-zlib) [ 100.929585] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 102.066576] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 103.863257] Key type lgssc registered [ 105.696191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 189.132385] hrtimer: interrupt took 5316964 ns [ 226.912919] Lustre: Mounted lustre-client [ 232.356559] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.435580] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing check_logdir /tmp/testlogs/ [ 252.383158] Lustre: lustre-OST0000-osc-ffff8e8252b7a800: disconnect after 23s idle [ 254.899242] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing yml_node [ 258.701663] Lustre: DEBUG MARKER: Client: 2.17.54.204 [ 261.271924] Lustre: DEBUG MARKER: MDS: 2.17.54.204 [ 263.934432] Lustre: DEBUG MARKER: OSS: 2.17.54.204 [ 265.709983] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Jul 28 02:22:50 EDT 2026 [ 280.876673] Lustre: Unmounted lustre-client [ 298.238300] Key type lgssc unregistered [ 298.511578] LNet: 8426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.516065] LNetError: 8426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 298.558136] LNet: Removed LNI 192.168.202.45@tcp [ 299.380579] Key type .llcrypt unregistered [ 299.383270] Key type ._llcrypt unregistered [ 328.244672] Key type ._llcrypt registered [ 328.247582] Key type .llcrypt registered [ 328.530983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 328.571774] alg: No test for adler32 (adler32-zlib) [ 329.583627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 329.658207] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 329.687789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 333.806668] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 334.177397] LNet: 9311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.186468] LNetError: 9311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 334.212980] LNet: Removed LNI 192.168.202.45@tcp [ 334.875421] Key type .llcrypt unregistered [ 334.882199] Key type ._llcrypt unregistered [ 342.253801] Key type ._llcrypt registered [ 342.256234] Key type .llcrypt registered [ 343.673171] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 02:24:08 (1785219848) [ 344.143851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 344.152118] alg: No test for adler32 (adler32-zlib) [ 345.132279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.232365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.261502] LNet: 9723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.292434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.353819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 345.372731] LNet: 9730:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 352.107875] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 02:24:17 (1785219857) [ 352.411878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.440826] LNet: 10308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.474425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.606664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 359.982953] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 02:24:24 (1785219864) [ 360.186246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.214213] LNet: 10891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.240949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.330433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 366.561301] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 02:24:31 (1785219871) [ 366.659582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.680691] LNet: 11474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.705792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.788885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 372.647571] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 02:24:37 (1785219877) [ 372.844379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.871360] LNet: 12065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.891722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.971663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 373.347563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 378.844961] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 02:24:44 (1785219884) [ 378.964535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.017848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.066368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 379.472898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 379.929690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 386.941631] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 02:24:51 (1785219891) [ 387.122862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.151660] LNet: 13249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.158043] LNet: 13249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 387.223467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.344257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 394.169325] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 02:24:58 (1785219898) [ 394.290586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.362936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.468031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 394.500598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 395.184716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 395.232745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 395.693771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 395.735606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.206350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 396.244713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 396.790882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 396.828582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 397.392851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 397.441398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.000732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 398.045520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.555711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 398.628094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 399.245768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 399.290218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 399.816923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 399.857562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 400.335269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 400.364858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 400.966392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 401.010612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 401.629957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 401.674899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 407.916783] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 02:25:12 (1785219912) [ 408.273222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.312102] LNet: 14519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.323156] LNet: 14519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 408.362934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.498601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 408.547025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 415.648316] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 02:25:20 (1785219920) [ 415.763300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.835554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.939972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 415.993789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 424.072975] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 02:25:28 (1785219928) [ 424.166310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.219270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.387215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 424.424042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 431.320700] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 02:25:36 (1785219936) [ 431.594564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.628519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.685622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 431.731655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 437.827323] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 02:25:42 (1785219942) [ 438.003371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.084848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.186585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 438.214566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 445.215926] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 02:25:50 (1785219950) [ 445.360748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.421385] LNet: 17445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.427822] LNet: 17445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 445.456840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.592509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 445.637644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 452.351482] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 02:25:56 (1785219956) [ 452.463660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.557244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.725628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.198150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 461.224775] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 02:26:05 (1785219965) [ 461.348903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.391090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.641829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.958419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.821035] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 02:26:13 (1785219973) [ 468.976755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.019351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.226967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.438355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.375611] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 02:26:21 (1785219981) [ 477.784820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.872602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.816143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.273930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 487.962532] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 02:26:32 (1785219992) [ 488.270590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.455692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.630639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.745590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 495.103189] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 02:26:40 (1785220000) [ 495.216302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.293911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.418829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.471254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 501.815775] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 02:26:46 (1785220006) [ 502.105570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.155414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.278912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 508.729301] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 02:26:53 (1785220013) [ 508.856445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.914125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.054245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 515.879937] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 02:27:00 (1785220020) [ 516.007499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.049711] LNet: 22882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 516.060799] LNet: 22882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 516.124627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.301912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 516.724368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 523.665875] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 02:27:08 (1785220028) [ 523.814252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.907497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.060393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 524.131789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 530.466827] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 02:27:15 (1785220035) [ 530.570195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.623235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.678685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 530.703697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 536.785344] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 02:27:21 (1785220041) [ 536.928544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.996099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.110614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 537.164962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 543.673694] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 02:27:28 (1785220048) [ 543.766090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.794089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.834807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 543.854368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 550.812964] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 02:27:35 (1785220055) [ 550.911484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.969453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.035156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 551.088536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 551.103800] LNetError: 25837:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 551.136844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 551.160456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 557.553724] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 02:27:42 (1785220062) [ 557.840400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.866421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.929829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6b2ypUzR5p [ 558.019434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6b2ypUzR5p [ 558.082471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6b2ypUzR5p [ 564.368277] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 02:27:49 (1785220069) [ 564.541307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.568218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.605571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 564.632981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 564.671175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 570.194207] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 02:27:55 (1785220075) [ 570.462789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.507981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.560810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 570.572570] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 571.855333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 573.121486] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 575.047229] Key type lgssc registered [ 576.166369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 579.861364] LNetError: 27598:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.45@tcp99 has no available nets [ 582.970715] Key type lgssc unregistered [ 583.401663] LNetError: 28706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 583.415702] LNet: Removed LNI 192.168.202.45@tcp [ 584.127679] Key type .llcrypt unregistered [ 584.129725] Key type ._llcrypt unregistered [ 591.461334] Key type ._llcrypt registered [ 591.463598] Key type .llcrypt registered [ 592.769412] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 02:28:17 (1785220097) [ 593.103951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 593.116160] alg: No test for adler32 (adler32-zlib) [ 594.064581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.162771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 597.495050] LNet: Message forwarding will be enabled in 2s [ 600.610443] LNet: Message forwarding enabled [ 601.164949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 601.193838] LNet: Message forwarding will stop in 2s to 6s [ 604.712736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 605.689519] LNet: Message forwarding disabled [ 607.575667] LNet: Message forwarding will be enabled in 2s [ 610.847274] LNet: Message forwarding enabled [ 614.858707] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 02:28:39 (1785220119) [ 614.925287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.970880] LNet: 30117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.217322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.286417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 622.148553] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 02:28:47 (1785220127) [ 622.306808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.346692] LNet: 30699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.385247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.489286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 622.519236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 622.545656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 622.570968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 622.597723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 622.623224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 622.648742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 622.697907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 622.746687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 622.776751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 622.808910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 622.843222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 622.871906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 622.903978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 622.942429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 622.970372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 623.007643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 623.057671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 623.092412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 623.117219] LNetError: 30746:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 629.416496] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 02:28:54 (1785220134) [ 629.640539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.671328] LNet: 31317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.691494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.833200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 636.400606] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 02:29:01 (1785220141) [ 638.466973] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg245-server [ 640.271358] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 02:29:05 (1785220145) [ 640.409818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.430773] LNet: 32244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.455088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.531563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.549350] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 641.468838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[46-50]@tcp [ 641.497404] LNet: 32380:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 641.557422] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.46@tcp is being used as a gateway but routing feature is not turned on [ 641.582701] LNetError: 32251:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.46@tcp has gone from up to down [ 642.598178] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.50@tcp is being used as a gateway but routing feature is not turned on [ 642.617251] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 643.624931] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.47@tcp is being used as a gateway but routing feature is not turned on [ 643.658108] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 645.674099] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.49@tcp is being used as a gateway but routing feature is not turned on [ 645.691548] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 648.877479] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 02:29:13 (1785220153) [ 649.771338] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.46@tcp is being used as a gateway but routing feature is not turned on [ 649.787637] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 650.667548] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg245-server [ 652.253979] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 02:29:17 (1785220157) [ 652.442410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.459969] LNet: 33317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.464449] LNetError: 33317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.484383] LNet: Removed LNI 192.168.202.45@tcp [ 652.500284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.548320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 652.573589] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 653.362230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[46-51/2]@tcp [ 653.398710] LNet: 33434:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 653.408592] LNet: 33434:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 653.432076] LNetError: 33321:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.50@tcp has gone from up to down [ 653.446743] LNetError: 33321:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 653.455174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[46-51/2]@tcp [ 660.314639] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 02:29:24 (1785220164) [ 660.422089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.469904] LNetError: 34020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.538275] LNet: Removed LNI 192.168.202.45@tcp [ 660.610721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.740495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 660.778892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.796711] LNet: 34041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.802560] LNet: 34041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 660.841520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.909119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 660.955629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.982741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.023699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 661.158797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.197255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.252380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 661.319892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.351093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.387380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 661.453322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.497936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.555249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 661.648961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.694210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.759188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 661.830397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.873521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.976298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 662.064729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.117871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.209864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 662.310793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.396422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.474740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 662.532818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.561765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.635644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 662.719697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.789824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.842823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.082067] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 02:29:34 (1785220174) [ 672.107381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg245-server [ 674.033516] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 02:29:38 (1785220178) [ 675.928375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg245-server [ 677.729794] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 02:29:42 (1785220182) [ 677.964080] LNet: 35429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.976598] LNet: 35429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 678.566508] Key type .llcrypt unregistered [ 678.568343] Key type ._llcrypt unregistered [ 685.232155] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 685.353918] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 685.357928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 685.360305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 687.827956] Key type ._llcrypt registered [ 687.830266] Key type .llcrypt registered [ 687.994616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 688.007213] alg: No test for adler32 (adler32-zlib) [ 689.042091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.122204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.161052] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 690.261566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 690.276750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 692.092828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 692.146387] LNet: Removed LNI 10.1.2.3@tcp [ 692.225214] LNet: 36147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.231639] LNetError: 36147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.245774] LNet: Removed LNI 192.168.202.45@tcp [ 692.824702] Key type .llcrypt unregistered [ 692.826559] Key type ._llcrypt unregistered [ 707.049816] Key type ._llcrypt registered [ 707.053314] Key type .llcrypt registered [ 708.908315] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 02:30:13 (1785220213) [ 709.542249] Key type .llcrypt unregistered [ 709.547405] Key type ._llcrypt unregistered [ 718.566271] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 719.025780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 719.034174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 719.042774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 721.653498] Key type ._llcrypt registered [ 721.655169] Key type .llcrypt registered [ 721.851498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 721.893291] alg: No test for adler32 (adler32-zlib) [ 722.825732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.901563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.926493] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 723.845471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 723.879341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 726.704096] LNet: 37643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.707873] LNetError: 37643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 726.714096] LNet: Removed LNI 10.1.2.3@tcp [ 726.740016] LNet: Removed LNI 192.168.202.45@tcp [ 727.236568] Key type .llcrypt unregistered [ 727.238753] Key type ._llcrypt unregistered [ 741.876194] Key type ._llcrypt registered [ 741.887731] Key type .llcrypt registered [ 743.283933] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 02:30:47 (1785220247) [ 743.991493] Key type .llcrypt unregistered [ 744.000363] Key type ._llcrypt unregistered [ 750.648310] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 750.753374] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 750.759332] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 750.762408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 752.662434] Key type ._llcrypt registered [ 752.673149] Key type .llcrypt registered [ 752.919402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 752.936440] alg: No test for adler32 (adler32-zlib) [ 753.930618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.995516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 754.035080] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 756.039346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.059967] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 758.376752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 758.397602] LNet: Removed LNI 10.1.2.3@tcp [ 758.415481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 758.476012] LNet: Removed LNI 10.1.2.31@tcp [ 758.486438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 758.515196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.332036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 759.369797] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 760.308524] LNet: 39559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.317722] LNetError: 39559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 760.322735] LNet: Removed LNI 10.1.2.31@tcp [ 760.339235] LNet: Removed LNI 10.1.2.3@tcp [ 761.035819] Key type .llcrypt unregistered [ 761.037415] Key type ._llcrypt unregistered [ 772.959303] Key type ._llcrypt registered [ 772.961342] Key type .llcrypt registered [ 774.227848] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 02:31:19 (1785220279) [ 774.595745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 774.619688] alg: No test for adler32 (adler32-zlib) [ 775.594322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.662308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 775.684710] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 776.555381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.567692] LNet: 40512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.572529] LNetError: 40512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.585939] LNet: Removed LNI 192.168.202.45@tcp [ 776.597745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.660333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.45@tcp [ 776.688628] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 777.395799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.45@tcp [ 777.423141] LNet: Removed LNI 192.168.202.45@tcp [ 777.490308] LNet: 40647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.224609] Key type .llcrypt unregistered [ 778.230283] Key type ._llcrypt unregistered [ 790.296058] Key type ._llcrypt registered [ 790.297827] Key type .llcrypt registered [ 791.699623] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 02:31:36 (1785220296) [ 794.005412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg245-server [ 795.698706] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 02:31:40 (1785220300) [ 796.237345] Key type .llcrypt unregistered [ 796.238991] Key type ._llcrypt unregistered [ 802.658634] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 802.898809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 802.914379] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 802.930336] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.563779] Key type ._llcrypt registered [ 805.566380] Key type .llcrypt registered [ 805.835814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 805.877457] alg: No test for adler32 (adler32-zlib) [ 806.840163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.919426] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 807.956885] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 810.215460] LNet: 42483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.221302] LNetError: 42483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.228546] LNet: Removed LNI 192.168.202.45@tcp1 [ 810.242631] LNet: Removed LNI 10.1.2.3@tcp2 [ 811.055416] Key type .llcrypt unregistered [ 811.061711] Key type ._llcrypt unregistered [ 825.264369] Key type ._llcrypt registered [ 825.268760] Key type .llcrypt registered [ 826.638406] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 02:32:11 (1785220331) [ 827.055409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.077272] alg: No test for adler32 (adler32-zlib) [ 828.130746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.239549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 828.297618] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 829.305861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.362193] LNet: 43452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.382553] LNetError: 43452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.397835] LNet: Removed LNI 192.168.202.45@tcp [ 829.420313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.505741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 829.551870] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 836.959712] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 02:32:21 (1785220341) [ 837.244947] LNet: 44044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.250320] LNetError: 44044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.266257] LNet: Removed LNI 192.168.202.45@tcp [ 837.930395] Key type .llcrypt unregistered [ 837.932622] Key type ._llcrypt unregistered [ 844.431589] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 844.666211] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 844.677738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 844.687747] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 847.444047] Key type ._llcrypt registered [ 847.445571] Key type .llcrypt registered [ 847.659054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 847.690805] alg: No test for adler32 (adler32-zlib) [ 848.643370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.697351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.742619] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 849.731298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 849.761598] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 852.312689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.328833] LNet: 44786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.334377] LNetError: 44786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 852.339321] LNet: Removed LNI 10.1.2.3@tcp [ 852.348612] LNet: Removed LNI 192.168.202.45@tcp [ 852.368482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.413951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 852.428791] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 852.431046] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.162826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.188603] LNet: 44813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.195968] LNetError: 44813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.201731] LNetError: 44813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 853.209929] LNet: Removed LNI 10.1.2.3@tcp [ 853.236539] LNet: Removed LNI 192.168.202.45@tcp [ 853.282174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.358682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 853.389659] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 853.404106] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 854.074362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 854.105153] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 855.249802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.288763] LNet: 44975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.293202] LNetError: 44975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 855.298249] LNetError: 44975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 855.302942] LNet: Removed LNI 10.1.2.3@tcp [ 855.306175] LNet: Removed LNI 192.168.202.45@tcp [ 855.335570] LNet: Removed LNI 192.168.202.45@tcp2 [ 855.360518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.426877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 855.495920] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 855.500986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 855.514267] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 863.939541] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 02:32:48 (1785220368) [ 864.189212] LNet: 45573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.194935] LNetError: 45573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 864.199598] LNetError: 45573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 864.204252] LNet: Removed LNI 10.1.2.3@tcp [ 864.208693] LNet: Removed LNI 192.168.202.45@tcp [ 864.223326] LNet: Removed LNI 192.168.202.45@tcp2 [ 864.772688] Key type .llcrypt unregistered [ 864.774219] Key type ._llcrypt unregistered [ 872.490523] Key type ._llcrypt registered [ 872.494449] Key type .llcrypt registered [ 872.752336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 872.781981] alg: No test for adler32 (adler32-zlib) [ 873.797883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.860731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.925559] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 874.792767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.812726] LNet: 45967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.818889] LNetError: 45967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.836331] LNet: Removed LNI 192.168.202.45@tcp [ 874.859436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.924498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 874.936707] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 875.565449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.579301] LNet: 45997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.583879] LNetError: 45997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.605937] LNet: Removed LNI 192.168.202.45@tcp [ 875.631260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.675627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 875.695403] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 876.264769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.292299] LNet: Removed LNI 192.168.202.45@tcp [ 876.314477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.423721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 876.494181] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 877.199820] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 877.419506] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 877.425665] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 877.428329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 879.886106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.933869] LNet: 46106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.941472] LNet: 46106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 879.950985] LNetError: 46106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.956480] LNetError: 46106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 879.981936] LNet: Removed LNI 192.168.202.45@tcp [ 880.016818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.105407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 880.139263] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 887.181765] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 02:33:12 (1785220392) [ 887.548627] LNet: 46703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.555580] LNetError: 46703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.569524] LNet: Removed LNI 192.168.202.45@tcp [ 888.112711] Key type .llcrypt unregistered [ 888.117883] Key type ._llcrypt unregistered [ 894.181724] Key type ._llcrypt registered [ 894.184402] Key type .llcrypt registered [ 894.476938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.511526] alg: No test for adler32 (adler32-zlib) [ 895.447485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.563925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.602270] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 896.767382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.787145] LNet: 47101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.792748] LNetError: 47101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.829832] LNet: Removed LNI 192.168.202.45@tcp [ 896.860916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.973704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 897.008911] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 903.576561] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 02:33:28 (1785220408) [ 903.801364] LNet: 47695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.808684] LNetError: 47695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.821136] LNet: Removed LNI 192.168.202.45@tcp [ 904.377528] Key type .llcrypt unregistered [ 904.378914] Key type ._llcrypt unregistered [ 910.581089] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.799167] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.887590] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.898537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 913.538973] Key type ._llcrypt registered [ 913.541721] Key type .llcrypt registered [ 913.741574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.751685] alg: No test for adler32 (adler32-zlib) [ 914.727835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.832368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.876435] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 915.832730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.847490] LNet: 48143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.853472] LNetError: 48143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.863863] LNet: Removed LNI 192.168.202.45@tcp [ 915.882066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.944771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 915.963274] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 922.275233] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 02:33:47 (1785220427) [ 922.450816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.468909] LNet: 48741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.473898] LNetError: 48741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.491087] LNet: Removed LNI 192.168.202.45@tcp [ 922.548507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.653852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 929.022040] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 02:33:53 (1785220433) [ 929.362154] LNet: 49324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.347848] Key type .llcrypt unregistered [ 930.351405] Key type ._llcrypt unregistered [ 937.590665] Key type ._llcrypt registered [ 937.597361] Key type .llcrypt registered [ 937.858855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.888074] alg: No test for adler32 (adler32-zlib) [ 938.825614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.931514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 938.967800] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 939.964934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.981586] LNet: 49721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.986386] LNetError: 49721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.002206] LNet: Removed LNI 192.168.202.45@tcp [ 940.041288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.133791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 940.161895] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 946.931088] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 02:34:11 (1785220451) [ 947.018746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.034336] LNet: 50307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.037375] LNetError: 50307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.065848] LNet: Removed LNI 192.168.202.45@tcp [ 947.089696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.186747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 954.002630] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 02:34:18 (1785220458) [ 955.647305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 957.088537] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 02:34:22 (1785220462) [ 958.745221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 960.506693] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 02:34:25 (1785220465) [ 962.171520] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 963.979474] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 02:34:28 (1785220468) [ 965.446859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 967.103281] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 02:34:32 (1785220472) [ 968.651494] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 970.639908] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 02:34:35 (1785220475) [ 970.749115] LNet: 52358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.410524] Key type .llcrypt unregistered [ 971.416259] Key type ._llcrypt unregistered [ 978.206962] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 978.458515] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 978.471293] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 978.484059] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 981.120649] Key type ._llcrypt registered [ 981.124451] Key type .llcrypt registered [ 981.405806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 981.436221] alg: No test for adler32 (adler32-zlib) [ 982.418691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.500839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.518396] LNet: Added LNI 192.168.202.45@tcp [8/512/0/180] [ 982.523206] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 982.525686] LNet: 52686:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 982.618535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.642100] LNet: 52705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.653259] LNetError: 52705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 982.663589] LNet: Removed LNI 10.1.2.3@tcp [ 982.665499] LNetError: 52705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.692795] LNet: Removed LNI 192.168.202.45@tcp [ 982.724573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.823962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.853255] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 982.941720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.975165] LNet: 52729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.984836] LNetError: 52729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.995455] LNet: Removed LNI 192.168.202.45@tcp [ 983.021600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.097486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 983.130125] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 983.135229] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 983.139325] LNet: 52737:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 983.242906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.269892] LNet: 52755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.275570] LNetError: 52755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 983.281482] LNet: Removed LNI 10.1.2.3@tcp [ 983.285413] LNetError: 52755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.296962] LNet: Removed LNI 192.168.202.45@tcp [ 983.320904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.418225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 983.448143] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 983.457456] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 983.467451] LNet: 52763:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 983.673796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.690389] LNet: 52789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.694990] LNetError: 52789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 983.700735] LNet: Removed LNI 10.1.2.3@tcp [ 983.703270] LNetError: 52789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.721314] LNet: Removed LNI 192.168.202.45@tcp [ 983.756247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.833897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 983.869963] LNet: Added LNI 192.168.202.45@tcp [8/256/0/182] [ 983.875499] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 984.018206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.064278] LNet: 52817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.069661] LNetError: 52817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 984.072826] LNet: Removed LNI 10.1.2.3@tcp [ 984.076508] LNetError: 52817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.093772] LNet: Removed LNI 192.168.202.45@tcp [ 984.144570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.297547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 984.339553] LNet: Added LNI 192.168.202.45@tcp [8/256/0/182] [ 984.346833] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 984.348503] LNet: 52825:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 984.438047] LNet: Removed LNI 10.1.2.3@tcp [ 984.464904] LNet: Removed LNI 192.168.202.45@tcp [ 985.114867] Key type .llcrypt unregistered [ 985.120527] Key type ._llcrypt unregistered [ 995.893937] Key type ._llcrypt registered [ 995.895944] Key type .llcrypt registered [ 997.123958] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 02:35:02 (1785220502) [ 997.696417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.718179] alg: No test for adler32 (adler32-zlib) [ 998.666647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.725973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.764590] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 999.553769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.45 [ 999.588670] LNet: 53800:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 999.600536] LNet: Removed LNI 192.168.202.45@tcp [ 1006.909444] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 02:35:11 (1785220511) [ 1007.046066] LNet: 54362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.059276] LNetError: 54362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.079318] LNet: Removed LNI 192.168.202.45@tcp [ 1007.806713] Key type .llcrypt unregistered [ 1007.808226] Key type ._llcrypt unregistered [ 1014.392948] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1014.725553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1016.411590] Key type ._llcrypt registered [ 1016.417587] Key type .llcrypt registered [ 1016.628893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1016.646713] alg: No test for adler32 (adler32-zlib) [ 1018.194424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.561412] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 02:35:29 (1785220529) [ 1024.648977] LNet: 55269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.657523] LNetError: 55269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.706135] LNet: Removed LNI 10.1.2.3@tcp [ 1025.346830] Key type .llcrypt unregistered [ 1025.352256] Key type ._llcrypt unregistered [ 1031.755181] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1031.955867] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1031.958955] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1031.962805] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1033.583196] Key type ._llcrypt registered [ 1033.585455] Key type .llcrypt registered [ 1033.739384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1033.754407] alg: No test for adler32 (adler32-zlib) [ 1034.747375] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1040.328113] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 02:35:45 (1785220545) [ 1041.708866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1043.126271] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 02:35:48 (1785220548) [ 1043.347880] LNet: 56454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.354328] LNetError: 56454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.365277] LNet: Removed LNI 10.1.2.3@tcp [ 1043.912350] Key type .llcrypt unregistered [ 1043.918988] Key type ._llcrypt unregistered [ 1048.961708] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1049.148510] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1049.152969] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1049.158815] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1051.798700] Key type ._llcrypt registered [ 1051.800801] Key type .llcrypt registered [ 1051.984992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.004484] alg: No test for adler32 (adler32-zlib) [ 1052.965237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.036671] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1061.434527] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 02:36:06 (1785220566) [ 1061.851429] LNet: 57682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.855771] LNetError: 57682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1061.869377] LNet: Removed LNI 10.1.2.3@tcp [ 1062.566688] Key type .llcrypt unregistered [ 1062.573089] Key type ._llcrypt unregistered [ 1069.229193] Key type ._llcrypt registered [ 1069.231796] Key type .llcrypt registered [ 1069.493408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1069.512734] alg: No test for adler32 (adler32-zlib) [ 1070.554166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1070.584296] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1078.212867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1078.802149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_interrupt -r 1 [ 1078.843435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_interrupt -r 1 [ 1078.871125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1078.884692] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1079.092876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.119920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1079.147901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.171359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.196705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.221900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.244841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.277250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1079.291620] LNet: 58120:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.307348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.331715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.353533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.375645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.401556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.423940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.450978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1079.895319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_dropped -r 1 [ 1079.922471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_dropped -r 1 [ 1079.962607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1079.984211] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1080.195658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1080.222507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1080.261927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.297518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1080.323981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.374164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.431932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.486895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1080.506447] LNet: 58229:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1080.530700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1080.563955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1080.590097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1080.617481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1080.662331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1080.698342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1080.733326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1081.359104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_aborted -r 1 [ 1081.433411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_aborted -r 1 [ 1081.498598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1081.526624] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1081.731064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1081.756351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1081.789907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1081.820793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1081.850915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1081.879960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1081.887209] LNet: 1 local NIs in recovery (showing 1): 192.168.202.45@tcp [ 1081.934566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1081.978442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1082.000346] LNet: 58339:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1082.034844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1082.092338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1082.141397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.196773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.251211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.287409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.340266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1082.837622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_no_route -r 1 [ 1082.889181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_no_route -r 1 [ 1082.925226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1083.177874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.211858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1083.250318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.304145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.350063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.397602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.440245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.500041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.526228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.541687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.581141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.623810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.662579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.707700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.773347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1084.281504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_timeout -r 1 [ 1084.304807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_timeout -r 1 [ 1084.333430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1084.345762] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1084.350119] LNet: Skipped 1 previous similar message [ 1084.527207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1084.549285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1084.570404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1084.592623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1084.613604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1084.636608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.656396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.678953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.693809] LNet: 58557:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.697771] LNet: 58557:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1084.710382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.739791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.763203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.785345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.811551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.839762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.866402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1085.349216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_error -r 1 [ 1085.391654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_error -r 1 [ 1085.427844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1085.625846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.664659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1085.699733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.750952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.796904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.832162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.861957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1085.906823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1085.934851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1085.973428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.020862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.056394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.087790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.118199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.161924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1086.398883] LNet: 58707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.407503] LNetError: 58707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.473326] LNet: Removed LNI 192.168.202.45@tcp [ 1087.873212] Key type .llcrypt unregistered [ 1087.876585] Key type ._llcrypt unregistered [ 1103.239374] Key type ._llcrypt registered [ 1103.240865] Key type .llcrypt registered [ 1104.390761] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 02:36:49 (1785220609) [ 1105.266301] Key type .llcrypt unregistered [ 1105.270379] Key type ._llcrypt unregistered [ 1110.653090] Key type ._llcrypt registered [ 1110.660611] Key type .llcrypt registered [ 1110.898513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.911612] alg: No test for adler32 (adler32-zlib) [ 1111.841475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1111.868251] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1119.782636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1130.799386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1130.833534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1130.863974] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1131.896962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_interrupt -r 1 [ 1131.928888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_interrupt -r 1 [ 1131.960525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_interrupt -r 1 [ 1131.986710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_interrupt -r 1 [ 1132.017269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_interrupt -r 1 [ 1132.061206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_interrupt -r 1 [ 1132.120754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_interrupt -r 1 [ 1132.168174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_interrupt -r 1 [ 1132.214542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1132.237549] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1132.251316] LNet: 1 local NIs in recovery (showing 1): 192.168.202.45@tcp [ 1132.466645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.489555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.510831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.534883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.557489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1132.582398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.598834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.622626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1132.636987] LNet: 60173:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1132.649683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1132.676624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1132.703862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.738642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.759753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.781623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.825703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1133.335621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_dropped -r 1 [ 1133.378369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_dropped -r 1 [ 1133.415605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_dropped -r 1 [ 1133.486933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_dropped -r 1 [ 1133.529281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_dropped -r 1 [ 1133.574436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_dropped -r 1 [ 1133.617315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_dropped -r 1 [ 1133.663759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_dropped -r 1 [ 1133.702872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1133.730789] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1133.735986] LNet: Skipped 2 previous similar messages [ 1133.955600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.988375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.014589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.039594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.064622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1134.091786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1134.120610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1134.147673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1134.171637] LNet: 60294:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1134.190898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1134.251587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1134.295806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1134.333767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1134.377450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1134.417395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.449937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1134.956678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_aborted -r 1 [ 1134.986527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_aborted -r 1 [ 1135.042328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_aborted -r 1 [ 1135.077211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_aborted -r 1 [ 1135.117869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_aborted -r 1 [ 1135.148315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_aborted -r 1 [ 1135.189883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_aborted -r 1 [ 1135.221379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_aborted -r 1 [ 1135.250279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1135.275969] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1135.279927] LNet: Skipped 2 previous similar messages [ 1135.538207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1135.596050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1135.646761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1135.687860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1135.735742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1135.778389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.809021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.840325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.855461] LNet: 60416:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1135.878651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.929987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.976522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.999818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.028608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1136.084271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1136.112847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1136.680331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_no_route -r 1 [ 1136.727081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_no_route -r 1 [ 1136.781497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_no_route -r 1 [ 1136.816179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_no_route -r 1 [ 1136.852818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_no_route -r 1 [ 1136.888740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_no_route -r 1 [ 1136.918197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_no_route -r 1 [ 1136.943598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_no_route -r 1 [ 1136.973761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1137.190167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.211709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.242631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.268709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.292358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.315939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.334893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.359120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.385864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.408740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.436322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.460344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.482886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.516533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.559723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1138.051063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_timeout -r 1 [ 1138.073858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_timeout -r 1 [ 1138.101732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_timeout -r 1 [ 1138.135874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_timeout -r 1 [ 1138.167608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_timeout -r 1 [ 1138.199406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_timeout -r 1 [ 1138.234912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_timeout -r 1 [ 1138.269545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_timeout -r 1 [ 1138.302170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1138.324729] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1138.330886] LNet: Skipped 5 previous similar messages [ 1138.556126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.596573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1138.629989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.650551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1138.675278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1138.694156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1138.711899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1138.732415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.742527] LNet: 60659:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1138.746981] LNet: 60659:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1138.765862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.801283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.834424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.880218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.914542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.942205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.966275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1139.458644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_error -r 1 [ 1139.489457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_error -r 1 [ 1139.526677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_error -r 1 [ 1139.565436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_error -r 1 [ 1139.636261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_error -r 1 [ 1139.689079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_error -r 1 [ 1139.739568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_error -r 1 [ 1139.796782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_error -r 1 [ 1139.836938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1140.025720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.071048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1140.109629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.190932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.225718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.271809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.315806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.346064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.389468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.447991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.492654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.530110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.564430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.607530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.668610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1146.889282] LNet: 60856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.902650] LNetError: 60856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1146.906466] LNet: Removed LNI 192.168.202.45@tcp [ 1147.942887] LNet: Removed LNI 192.168.202.45@tcp1 [ 1148.498771] Key type .llcrypt unregistered [ 1148.506278] Key type ._llcrypt unregistered [ 1163.925749] Key type ._llcrypt registered [ 1163.930637] Key type .llcrypt registered [ 1164.974628] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:37:50 (1785220670) [ 1165.721373] Key type .llcrypt unregistered [ 1165.724198] Key type ._llcrypt unregistered [ 1171.158217] Key type ._llcrypt registered [ 1171.161326] Key type .llcrypt registered [ 1171.388142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.435108] alg: No test for adler32 (adler32-zlib) [ 1172.373174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.393283] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1178.345795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1178.842930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e remote_dropped -r 1 [ 1178.874876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e remote_dropped -r 1 [ 1178.900278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1178.917755] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1179.084863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.131701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.170234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.208882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.250738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.293674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.321664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.347736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.361681] LNet: 62104:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.373690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.393189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.424255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.447639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.477709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.517665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.556595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1180.038272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e remote_error -r 1 [ 1180.067485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e remote_error -r 1 [ 1180.104742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1180.128265] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1180.379970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.412553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.441758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.458626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.473107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.490623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.511752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.530969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.543778] LNet: 62213:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.559492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.578665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.596588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.614930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.635647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.657466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.673852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1181.176572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e remote_timeout -r 1 [ 1181.232258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e remote_timeout -r 1 [ 1181.272699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1181.291750] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1181.617103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.661311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.711747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.747889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.793756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.829306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.877683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.911291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.924345] LNet: 62322:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.942284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.969504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.994198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.016783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.039440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.075147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.103770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1182.389332] LNet: 62363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.395337] LNetError: 62363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.410284] LNet: Removed LNI 192.168.202.45@tcp [ 1183.028541] Key type .llcrypt unregistered [ 1183.032343] Key type ._llcrypt unregistered [ 1200.092587] Key type ._llcrypt registered [ 1200.096043] Key type .llcrypt registered [ 1201.247643] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:38:26 (1785220706) [ 1201.893455] Key type .llcrypt unregistered [ 1201.895148] Key type ._llcrypt unregistered [ 1207.951839] Key type ._llcrypt registered [ 1207.954556] Key type .llcrypt registered [ 1208.126414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1208.138973] alg: No test for adler32 (adler32-zlib) [ 1209.059878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1209.089811] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1216.412604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1225.951585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1225.995203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1226.012170] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1227.177763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e remote_dropped -r 1 [ 1227.212549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e remote_dropped -r 1 [ 1227.243450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e remote_dropped -r 1 [ 1227.269945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e remote_dropped -r 1 [ 1227.299178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e remote_dropped -r 1 [ 1227.325118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e remote_dropped -r 1 [ 1227.348740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e remote_dropped -r 1 [ 1227.378188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e remote_dropped -r 1 [ 1227.406214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1227.420707] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1227.426850] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.145@tcp [ 1227.612972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.637732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1227.653578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.678600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.701212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.726416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.797289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.862734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.876805] LNet: 63828:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.893879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.921669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.948385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.001506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.051186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.140557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.189623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1228.225325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.758704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e remote_error -r 1 [ 1228.792436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e remote_error -r 1 [ 1228.829246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e remote_error -r 1 [ 1228.864216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e remote_error -r 1 [ 1228.896970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e remote_error -r 1 [ 1228.925687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e remote_error -r 1 [ 1228.953089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e remote_error -r 1 [ 1229.009036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e remote_error -r 1 [ 1229.055318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1229.084947] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1229.090469] LNet: Skipped 2 previous similar messages [ 1229.292339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.308511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.334678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.372302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.406128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.444059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.485190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.508593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.528454] LNet: 63951:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.543520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.562819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.596856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.646294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.688527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.725990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.760402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1229.791834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.249888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e remote_timeout -r 1 [ 1230.287777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e remote_timeout -r 1 [ 1230.314916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e remote_timeout -r 1 [ 1230.365522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e remote_timeout -r 1 [ 1230.424759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e remote_timeout -r 1 [ 1230.476123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e remote_timeout -r 1 [ 1230.519138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e remote_timeout -r 1 [ 1230.557993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e remote_timeout -r 1 [ 1230.599653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.145@tcp [ 1230.615773] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1230.897944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.955711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1231.039527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1231.095346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1231.128086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1231.161771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1231.187866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1231.220726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1231.253391] LNet: 64075:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1231.267962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1231.335809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.376283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1231.399766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1231.423294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.468684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.496637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1231.538153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.129170] LNet: 64153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.164753] LNetError: 64153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.177790] LNet: Removed LNI 192.168.202.45@tcp [ 1236.199889] LNet: Removed LNI 192.168.202.45@tcp1 [ 1236.677669] Key type .llcrypt unregistered [ 1236.680718] Key type ._llcrypt unregistered [ 1253.056610] Key type ._llcrypt registered [ 1253.057925] Key type .llcrypt registered [ 1254.103562] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:39:19 (1785220759) [ 1255.479614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1257.002892] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:39:22 (1785220762) [ 1257.815520] Key type .llcrypt unregistered [ 1257.819146] Key type ._llcrypt unregistered [ 1263.844959] Key type ._llcrypt registered [ 1263.847597] Key type .llcrypt registered [ 1264.022537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1264.037231] alg: No test for adler32 (adler32-zlib) [ 1264.918594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1264.940404] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1271.652940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1272.195978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e network_timeout -r 1 [ 1272.210688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e network_timeout -r 1 [ 1272.240829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1272.279129] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1272.489545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1272.525346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1272.547652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1272.572637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1272.603341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1272.624877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1272.643695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1272.669756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1272.692435] LNet: 65712:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1272.712660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1272.745928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1272.806399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1272.833893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1272.855974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1272.879821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.086953] LNet: 65751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.092432] LNetError: 65751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.158481] LNet: Removed LNI 192.168.202.45@tcp [ 1274.567559] Key type .llcrypt unregistered [ 1274.569630] Key type ._llcrypt unregistered [ 1292.374759] Key type ._llcrypt registered [ 1292.376699] Key type .llcrypt registered [ 1292.522241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1292.538654] alg: No test for adler32 (adler32-zlib) [ 1293.418400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1293.439542] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1299.949193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1307.601468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1307.642968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1307.661551] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1308.620505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e network_timeout -r 1 [ 1308.650459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e network_timeout -r 1 [ 1308.681840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e network_timeout -r 1 [ 1308.714104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e network_timeout -r 1 [ 1308.739425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e network_timeout -r 1 [ 1308.783641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e network_timeout -r 1 [ 1308.837497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e network_timeout -r 1 [ 1308.882310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e network_timeout -r 1 [ 1308.917546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1308.931521] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1309.099560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1309.120622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1309.137742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1309.161992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1309.183313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1309.207370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1309.230900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1309.245253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1309.258474] LNet: 66630:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1309.282349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1309.304896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1309.344830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1309.386412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1309.414648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1309.445514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1312.397775] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1312.633957] LNet: 66704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.639419] LNetError: 66704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.695398] LNet: Removed LNI 192.168.202.45@tcp [ 1313.697248] LNetError: 66704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.717595] LNet: Removed LNI 192.168.202.45@tcp1 [ 1314.115446] Key type .llcrypt unregistered [ 1314.119400] Key type ._llcrypt unregistered [ 1327.168146] Key type ._llcrypt registered [ 1327.169775] Key type .llcrypt registered [ 1328.133846] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:40:33 (1785220833) [ 1328.418413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.428383] alg: No test for adler32 (adler32-zlib) [ 1329.296534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.336579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1329.347930] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1329.854706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1329.867823] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1330.473983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 1330.522795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1330.557334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1330.577391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1330.603248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1330.629180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1330.657509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.45@tcp [ 1330.674347] LNetError: 67805:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.45@tcp to recovery [ 1331.359333] LNet: 1 local NIs in recovery (showing 1): 192.168.202.45@tcp [ 1331.359909] LNet: There was an unexpected network error while writing to 192.168.202.45: rc = -22 [ 1333.407416] LNet: There was an unexpected network error while writing to 192.168.202.45: rc = -22 [ 1334.081921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1337.503367] LNet: There was an unexpected network error while writing to 192.168.202.45: rc = -22 [ 1338.592073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1338.628428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1338.689788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1338.715560] LNet: 67922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.723456] LNetError: 67922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.727785] LNet: Removed LNI 192.168.202.45@tcp [ 1338.746343] LNet: Removed LNI 192.168.202.45@tcp1 [ 1338.758845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1338.812187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1338.847132] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1339.491110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1339.506891] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1340.351635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 1340.382859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1340.445334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1340.483708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1340.522148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1340.557939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1340.595984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1340.624224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.45@tcp [ 1340.649334] LNetError: 68195:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.45@tcp to recovery [ 1340.832508] LNet: There was an unexpected network error while writing to 192.168.202.45: rc = -22 [ 1343.039800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1346.976360] LNet: There was an unexpected network error while writing to 192.168.202.45: rc = -22 [ 1346.986940] LNet: Skipped 1 previous similar message [ 1351.254639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1351.292195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1351.330674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1357.417620] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:41:02 (1785220862) [ 1357.660252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.680422] LNet: 68903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.687779] LNetError: 68903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.702520] LNetError: 68903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1357.710307] LNet: Removed LNI 192.168.202.45@tcp [ 1357.728075] LNet: Removed LNI 192.168.202.45@tcp1 [ 1357.752741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.809996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1357.829725] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1358.525420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1358.546936] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1359.377982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 1359.417619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1359.436981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1359.474178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1359.512650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1359.541970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1359.572157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.45@tcp --health 0 [ 1359.639320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1359.839154] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.45@tcp [ 1360.863600] LNet: There was an unexpected network error while writing to 192.168.202.45: rc = -22 [ 1360.872789] LNet: Skipped 1 previous similar message [ 1376.065882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1376.199169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1376.232854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.258157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.45@tcp --health 500 [ 1376.364799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1382.932347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1382.987835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1383.012213] LNet: 69403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.027812] LNetError: 69403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.032016] LNetError: 69403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1384.095111] LNet: Removed LNI 192.168.202.45@tcp [ 1384.103749] LNet: Removed LNI 192.168.202.45@tcp1 [ 1384.121208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1384.170352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1384.192955] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1384.750169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1384.763671] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1385.450398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 1385.479597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1385.558530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1385.598651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1385.635380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1385.662244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1385.710825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1385.749181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.45@tcp --health 0 [ 1386.208702] LNet: There was an unexpected network error while writing to 192.168.202.45: rc = -22 [ 1386.221796] LNet: Skipped 3 previous similar messages [ 1387.033064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1397.099152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1397.126493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1397.149469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1402.960294] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:41:48 (1785220908) [ 1403.444843] LNet: 70449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.447801] LNetError: 70449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.451648] LNetError: 70449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1403.454858] LNet: Removed LNI 192.168.202.45@tcp [ 1403.463749] LNet: Removed LNI 192.168.202.45@tcp1 [ 1403.995952] Key type .llcrypt unregistered [ 1404.001923] Key type ._llcrypt unregistered [ 1410.200488] Key type ._llcrypt registered [ 1410.209699] Key type .llcrypt registered [ 1410.496461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1410.511817] alg: No test for adler32 (adler32-zlib) [ 1411.400259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1411.442433] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1419.470893] Lustre: DEBUG MARKER: Initial discovery [ 1419.671413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.145@tcp [ 1421.473849] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1421.585741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1421.615274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.145@tcp [ 1421.629130] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1421.647933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1421.696526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.145@tcp -r 1 -m GET -l 3 [ 1421.730504] LNet: 1 local NIs in recovery (showing 1): 192.168.202.45@tcp [ 1421.743348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.145@tcp -r 1 -m GET -e local_error [ 1421.775148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.145@tcp -r 1 -m PUT -l 6 [ 1422.778328] Lustre: DEBUG MARKER: Force oleg245-client.virtnet to discover 192.168.202.145@tcp (in background) [ 1422.832505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.145@tcp [ 1423.877431] Lustre: DEBUG MARKER: Force oleg245-server to discover 192.168.202.45@tcp [ 1425.450846] Lustre: DEBUG MARKER: Wait for 71207 [ 1425.890319] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1427.017951] Lustre: DEBUG MARKER: Finished wait on 71207 [ 1427.145863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1427.198044] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1427.210708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1427.394602] LNet: 71662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.398770] LNetError: 71662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.416791] LNet: Removed LNI 192.168.202.45@tcp [ 1427.682574] Key type .llcrypt unregistered [ 1427.684519] Key type ._llcrypt unregistered [ 1441.366592] Key type ._llcrypt registered [ 1441.368796] Key type .llcrypt registered [ 1442.472590] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:42:27 (1785220947) [ 1442.789114] Key type .llcrypt unregistered [ 1442.790572] Key type ._llcrypt unregistered [ 1448.271796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1448.370905] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1448.379669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1448.394842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1451.082425] Key type ._llcrypt registered [ 1451.084804] Key type .llcrypt registered [ 1451.306814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1451.336800] alg: No test for adler32 (adler32-zlib) [ 1452.274661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.347456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1452.388904] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1453.109423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1453.123967] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1460.845094] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:42:46 (1785220966) [ 1461.098072] LNet: 73741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.106736] LNetError: 73741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1461.114863] LNet: Removed LNI 10.1.2.3@tcp [ 1461.128071] LNet: Removed LNI 192.168.202.45@tcp [ 1461.675367] Key type .llcrypt unregistered [ 1461.679339] Key type ._llcrypt unregistered [ 1467.312290] Key type ._llcrypt registered [ 1467.313992] Key type .llcrypt registered [ 1467.550254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1467.576668] alg: No test for adler32 (adler32-zlib) [ 1468.566845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.634894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1468.671215] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1469.322489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1469.361688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1470.226690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.45@tcp [ 1470.267993] LNetError: Unexpected error -22 connecting to 192.168.202.45@tcp at host 192.168.202.45:988 [ 1475.488973] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:43:00 (1785220980) [ 1475.655399] LNet: 74867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.670729] LNetError: 74867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1475.688975] LNet: Removed LNI 10.1.2.3@tcp [ 1475.732347] LNet: Removed LNI 192.168.202.45@tcp [ 1476.196071] Key type .llcrypt unregistered [ 1476.199950] Key type ._llcrypt unregistered [ 1481.128996] Key type ._llcrypt registered [ 1481.131831] Key type .llcrypt registered [ 1481.290392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1481.307032] alg: No test for adler32 (adler32-zlib) [ 1482.200841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.238947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1482.251487] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1482.822518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1482.839790] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 1483.671730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.45@tcp1 --nid 192.168.202.45@tcp2 [ 1487.295850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.45@tcp1 192.168.202.45@tcp2 [ 1487.307710] LNetError: 75707:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.45@tcp1 to 192.168.202.45@tcp2 - Route table may be misconfigured [ 1487.331327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.45@tcp2 192.168.202.45@tcp1 [ 1492.301908] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:43:17 (1785220997) [ 1492.425362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.440642] LNet: 76280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.451088] LNetError: 76280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.455749] LNet: Removed LNI 192.168.202.45@tcp1 [ 1493.483151] LNet: Removed LNI 192.168.202.45@tcp2 [ 1493.521971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.571489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.588312] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1494.125323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1494.144278] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1494.779811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 1494.851240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.45@tcp 192.168.202.45@tcp [ 1494.888876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.45@tcp1 192.168.202.45@tcp1 [ 1494.937993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.45@tcp -d 192.168.202.45@tcp -e network_timeout [ 1494.972577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -e network_timeout [ 1494.995478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -e network_timeout [ 1495.018691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -e network_timeout [ 1495.064608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1495.082232] LNet: There was an unexpected network error while writing to 192.168.202.45: rc = -22 [ 1495.094660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1495.191834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1495.282406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1495.585584] LNet: 1 local NIs in recovery (showing 1): 192.168.202.45@tcp [ 1500.297535] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:43:25 (1785221005) [ 1500.418550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.454594] LNet: 77137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.460128] LNetError: 77137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.469837] LNetError: 77137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1500.476539] LNet: Removed LNI 192.168.202.45@tcp [ 1501.549953] LNet: Removed LNI 192.168.202.45@tcp1 [ 1501.619107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.716353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1501.931212] LNet: 77180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.264693] Key type .llcrypt unregistered [ 1502.268897] Key type ._llcrypt unregistered [ 1510.682434] Key type ._llcrypt registered [ 1510.683817] Key type .llcrypt registered [ 1511.682914] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:43:36 (1785221016) [ 1512.073098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1512.089992] alg: No test for adler32 (adler32-zlib) [ 1512.976927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1514.015969] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1515.232894] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:43:40 (1785221020) [ 1515.339120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1515.373172] LNet: 78277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.391691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.429118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1515.446384] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1515.871280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1515.884713] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1516.347101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1516.361593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp1 [ 1516.388689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp1 [ 1519.928508] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:43:45 (1785221025) [ 1521.046930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg245-server [ 1522.110058] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:43:47 (1785221027) [ 1523.389266] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg245-server [ 1524.635434] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:43:49 (1785221029) [ 1526.018575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg245-server [ 1527.400818] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:43:52 (1785221032) [ 1528.705964] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg245-server [ 1529.861128] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:43:55 (1785221035) [ 1531.126409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg245-server [ 1532.445965] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:43:57 (1785221037) [ 1533.874742] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg245-server [ 1535.068789] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:44:00 (1785221040) [ 1536.401237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg245-server [ 1537.496743] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:44:02 (1785221042) [ 1538.704896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg245-server [ 1539.581415] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:44:05 (1785221045) [ 1540.702541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg245-server [ 1542.079682] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:44:07 (1785221047) [ 1542.152434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.183804] LNet: 82145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.192814] LNetError: 82145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.204743] LNet: Removed LNI 192.168.202.45@tcp [ 1543.269695] LNet: Removed LNI 192.168.202.45@tcp1 [ 1543.293185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.334262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.364909] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1544.047637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1544.118618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1544.232818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1544.249186] LNet: 82300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.253701] LNetError: 82300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.256512] LNetError: 82300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1545.317565] LNet: Removed LNI 192.168.202.45@tcp [ 1545.341401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.416109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1545.439163] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1545.994879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1546.115688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.144127] LNetError: 82445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.173884] LNet: Removed LNI 192.168.202.45@tcp [ 1546.199317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.260414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.288148] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1546.776547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1546.836214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1546.955172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.968814] LNet: 82592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.973650] LNet: 82592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1548.005935] LNet: Removed LNI 192.168.202.45@tcp [ 1548.032434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.089795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1548.112725] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1548.747454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1548.945573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.968302] LNetError: 82738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.975064] LNetError: 82738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1549.000823] LNet: Removed LNI 192.168.202.45@tcp [ 1549.027523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.128275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.151339] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1549.824322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1549.892485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1550.022291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.093838] LNet: Removed LNI 192.168.202.45@tcp [ 1551.165599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.246498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1551.262845] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1551.923623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1552.141928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.209576] LNet: 83030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.218828] LNet: 83030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1553.253580] LNet: Removed LNI 192.168.202.45@tcp [ 1553.278306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.383564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1553.435327] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1554.137380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1554.205660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1555.546421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.570747] LNetError: 83186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.574030] LNetError: 83186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1555.619244] LNet: Removed LNI 192.168.202.45@tcp [ 1555.657580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.724450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1555.741645] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1556.252839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1556.438611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.478163] LNet: Removed LNI 192.168.202.45@tcp [ 1557.505763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.579840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1557.602211] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1558.240459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1558.319491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1559.665209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.741663] LNet: Removed LNI 192.168.202.45@tcp [ 1560.763251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.831175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1560.856070] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1561.445961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1561.611726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.631153] LNet: 83633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.636740] LNet: 83633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1562.664401] LNet: Removed LNI 192.168.202.45@tcp [ 1562.692132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.729664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.752662] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1563.322366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1563.388275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1563.530660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.584485] LNet: Removed LNI 192.168.202.45@tcp [ 1564.624937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.673465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1564.702244] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1565.316566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1565.457303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.473390] LNetError: 83926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1565.476434] LNetError: 83926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1566.502994] LNet: Removed LNI 192.168.202.45@tcp [ 1566.535779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.618720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1566.665528] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1567.182277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1567.231152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.256342] LNet: Removed LNI 192.168.202.45@tcp [ 1567.277047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.322517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1567.340794] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1567.880342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1567.943828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.966743] LNet: Removed LNI 192.168.202.45@tcp [ 1567.983809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.040221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1568.132872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.169622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.219746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1568.290715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.320531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.360970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1568.421843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.453672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.505497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1568.552374] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1569.196741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.235686] LNet: Removed LNI 192.168.202.45@tcp [ 1569.259282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.310238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1574.787241] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:44:39 (1785221079) [ 1574.897253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.955428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.015540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1575.036720] LNet: Added LNI 192.168.202.45@tcp231 [8/256/0/180] [ 1575.987786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.016330] LNet: Removed LNI 192.168.202.45@tcp231 [ 1576.036676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.097508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1576.115994] LNet: Added LNI 192.168.202.45@tcp231 [8/256/0/180] [ 1576.123737] LNet: 85093:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1576.546254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1576.583971] LNet: Removed LNI 192.168.202.45@tcp231 [ 1576.606824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1576.638596] LNet: Added LNI 192.168.202.45@tcp231 [8/256/0/180] [ 1577.536351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.555795] LNet: Removed LNI 192.168.202.45@tcp231 [ 1577.568829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.631544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1577.651982] LNet: Added LNI 192.168.202.45@tcp231 [8/256/0/0] [ 1577.655085] LNet: 85245:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.990039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1578.024975] LNet: Removed LNI 192.168.202.45@tcp231 [ 1578.042414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1578.061962] LNet: Added LNI 192.168.202.45@tcp231 [8/256/0/0] [ 1578.877880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.894054] LNet: 85388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.898450] LNet: 85388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1578.906283] LNet: Removed LNI 192.168.202.45@tcp231 [ 1578.922943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.978485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1578.999052] LNet: Added LNI 192.168.202.45@tcp231 [8/256/0/60] [ 1579.004434] LNet: 85396:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1579.306690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1579.346601] LNet: Removed LNI 192.168.202.45@tcp231 [ 1579.367754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1579.387371] LNet: Added LNI 192.168.202.45@tcp231 [8/256/0/60] [ 1580.238899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.275802] LNet: Removed LNI 192.168.202.45@tcp231 [ 1580.293678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.343398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1580.360446] LNet: Added LNI 192.168.202.45@tcp231 [8/256/0/180] [ 1580.924679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1580.973801] LNet: Removed LNI 192.168.202.45@tcp231 [ 1580.984398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1581.004556] LNet: Added LNI 192.168.202.45@tcp231 [8/256/0/180] [ 1586.807452] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:44:52 (1785221092) [ 1586.936886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.956099] LNetError: 86257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.961566] LNetError: 86257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1586.975579] LNet: Removed LNI 192.168.202.45@tcp231 [ 1586.989492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.046889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1587.073818] LNet: Added LNI 192.168.202.45@tcp232 [8/256/0/180] [ 1587.604808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1587.627731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1587.645619] LNet: Removed LNI 192.168.202.45@tcp232 [ 1587.655559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1587.673489] LNet: Added LNI 192.168.202.45@tcp232 [8/256/0/180] [ 1588.140285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1588.170171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1588.197617] LNet: Removed LNI 192.168.202.45@tcp232 [ 1588.209855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1588.230637] LNet: Added LNI 192.168.202.45@tcp232 [8/256/0/180] [ 1588.821894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1588.881656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1588.912000] LNet: Removed LNI 192.168.202.45@tcp232 [ 1588.940740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1588.965922] LNet: Added LNI 192.168.202.45@tcp232 [8/256/0/180] [ 1589.468149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1589.504587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1589.532800] LNet: Removed LNI 192.168.202.45@tcp232 [ 1589.546613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1593.970653] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:44:59 (1785221099) [ 1594.780309] Key type .llcrypt unregistered [ 1594.782222] Key type ._llcrypt unregistered [ 1599.294684] Key type ._llcrypt registered [ 1599.295765] Key type .llcrypt registered [ 1599.455520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1599.477769] alg: No test for adler32 (adler32-zlib) [ 1600.332958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1600.353613] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1605.632966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1612.244755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.284342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1612.299507] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1613.540601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_interrupt -r 2 [ 1613.578772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_interrupt -r 2 [ 1613.613258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_interrupt -r 2 [ 1613.647869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_interrupt -r 2 [ 1613.691512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_interrupt -r 2 [ 1613.732818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_interrupt -r 2 [ 1613.764582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_interrupt -r 2 [ 1613.813267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_interrupt -r 2 [ 1613.880923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.45@tcp 192.168.202.145@tcp [ 1613.915528] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1613.927055] LNet: 1 local NIs in recovery (showing 1): 192.168.202.45@tcp [ 1614.167157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1614.199406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1614.225445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1614.261500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1614.311296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1614.355274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1614.389301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1614.434709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1614.458493] LNet: 88047:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1614.468344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1614.488279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1614.530635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1614.554446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1614.577527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1614.611815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1615.120902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_dropped -r 2 [ 1615.162640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_dropped -r 2 [ 1615.201637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_dropped -r 2 [ 1615.230676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_dropped -r 2 [ 1615.259661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_dropped -r 2 [ 1615.307455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_dropped -r 2 [ 1615.352948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_dropped -r 2 [ 1615.385519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_dropped -r 2 [ 1615.416559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.45@tcp 192.168.202.145@tcp [ 1615.434620] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1615.695686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1615.748482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1615.776792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1615.802206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1615.841189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1615.877242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1615.902590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1615.938840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1615.962280] LNet: 88167:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1615.992703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1616.022379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1616.053199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1616.095394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1616.122549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1616.153627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1616.590634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_aborted -r 2 [ 1616.621724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_aborted -r 2 [ 1616.649213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_aborted -r 2 [ 1616.677724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_aborted -r 2 [ 1616.711404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_aborted -r 2 [ 1616.746922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_aborted -r 2 [ 1616.772911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_aborted -r 2 [ 1616.804151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_aborted -r 2 [ 1616.836479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.45@tcp 192.168.202.145@tcp [ 1616.865604] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1617.032615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1617.052043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1617.075735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1617.099855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1617.131254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1617.162474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.188382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.217170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1617.230841] LNet: 88287:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1617.246631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1617.270708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1617.297714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1617.322394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1617.344456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1617.368313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1617.830124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_no_route -r 2 [ 1617.857574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_no_route -r 2 [ 1617.888430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_no_route -r 2 [ 1617.914785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_no_route -r 2 [ 1617.941840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_no_route -r 2 [ 1617.970118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_no_route -r 2 [ 1618.014675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_no_route -r 2 [ 1618.045560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_no_route -r 2 [ 1618.081437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.45@tcp 192.168.202.145@tcp [ 1618.282343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1618.307279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1618.326787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1618.344136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1618.372347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1618.392782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1618.416705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1618.446625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1618.478974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1618.502686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1618.531244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1618.560282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1618.591411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.624204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1619.098774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e local_timeout -r 2 [ 1619.129192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e local_timeout -r 2 [ 1619.178213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e local_timeout -r 2 [ 1619.215386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e local_timeout -r 2 [ 1619.276544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e local_timeout -r 2 [ 1619.322492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e local_timeout -r 2 [ 1619.368388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e local_timeout -r 2 [ 1619.407707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e local_timeout -r 2 [ 1619.456853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.45@tcp 192.168.202.145@tcp [ 1619.478059] LNet: There was an unexpected network error while writing to 192.168.202.145: rc = -22 [ 1619.483726] LNet: Skipped 1 previous similar message [ 1619.791964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1619.818341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1619.840973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1619.864112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1619.880483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1619.907897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1619.932428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1619.972777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1619.986539] LNet: 88528:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1619.990796] LNet: 88528:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1620.009868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1620.033250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1620.053503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1620.072850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1620.097313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1620.123245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1620.570635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp -m GET -e remote_dropped -r 2 [ 1620.617230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp -d 192.168.202.145@tcp1 -m GET -e remote_dropped -r 2 [ 1620.699330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp -m GET -e remote_dropped -r 2 [ 1620.754819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp -d 192.168.202.45@tcp1 -m GET -e remote_dropped -r 2 [ 1620.797856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp -m GET -e remote_dropped -r 2 [ 1620.845634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.145@tcp1 -m GET -e remote_dropped -r 2 [ 1620.875554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp -m GET -e remote_dropped -r 2 [ 1620.902452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.45@tcp1 -d 192.168.202.45@tcp1 -m GET -e remote_dropped -r 2 [ 1620.924945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.45@tcp 192.168.202.145@tcp [ 1620.936789] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.145@tcp [ 1621.105638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1621.130576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1621.152380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1621.183510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1621.207654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1621.238319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1621.258440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1621.277226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1621.314336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1621.345352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1621.370058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1621.404890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1621.437431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1621.463665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1625.305675] LNet: 88722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.309684] LNetError: 88722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.321132] LNet: Removed LNI 192.168.202.45@tcp [ 1625.332739] LNet: Removed LNI 192.168.202.45@tcp1 [ 1625.752604] Key type .llcrypt unregistered [ 1625.754914] Key type ._llcrypt unregistered [ 1639.004727] Key type ._llcrypt registered [ 1639.007762] Key type .llcrypt registered [ 1640.026407] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:45:45 (1785221145) [ 1640.247339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1640.256915] alg: No test for adler32 (adler32-zlib) [ 1641.147396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.187463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.206167] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1642.035422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1642.050404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.45@tcp [ 1642.070418] LNetError: 89701:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.45@tcp to recovery [ 1642.136867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.45@tcp [ 1642.209619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1647.244962] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:45:52 (1785221152) [ 1647.459697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.479704] LNet: 90291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.485286] LNetError: 90291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.503791] LNet: Removed LNI 192.168.202.45@tcp [ 1647.541463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.594398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1647.613085] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1648.210542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1648.236122] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1648.889674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.45@tcp [ 1648.907982] LNetError: 90557:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.45@tcp to recovery [ 1649.631561] LNet: 1 local NIs in recovery (showing 1): 192.168.202.45@tcp [ 1649.633133] LNet: There was an unexpected network error while writing to 192.168.202.45: rc = -22 [ 1650.133442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.45@tcp [ 1657.137409] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:46:02 (1785221162) [ 1658.612244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg245-server [ 1659.860593] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:46:05 (1785221165) [ 1661.439754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg245-server [ 1662.839730] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:46:08 (1785221168) [ 1664.427906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg245-server [ 1665.993141] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:46:11 (1785221171) [ 1666.075956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.102170] LNet: 92182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.104815] LNetError: 92182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.167439] LNet: Removed LNI 192.168.202.45@tcp [ 1667.169400] LNetError: 92182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.198465] LNet: Removed LNI 192.168.202.45@tcp1 [ 1667.251540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.374324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1667.426496] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1668.199567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1668.274533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1668.355520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1668.473688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1668.519926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1668.560833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1673.958966] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:46:19 (1785221179) [ 1674.145278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.191650] LNet: 92913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.197868] LNetError: 92913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.218685] LNet: Removed LNI 192.168.202.45@tcp [ 1674.238669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.295909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1674.340991] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1680.478498] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:46:25 (1785221185) [ 1680.666854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.693585] LNet: 93617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.701175] LNetError: 93617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.714721] LNet: Removed LNI 192.168.202.45@tcp [ 1680.738789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.799368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1680.816313] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1686.961530] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:46:32 (1785221192) [ 1688.066814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1689.739881] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:46:34 (1785221194) [ 1690.284787] LNet: 94652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.291457] LNetError: 94652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.309561] LNet: Removed LNI 192.168.202.45@tcp [ 1691.016814] Key type .llcrypt unregistered [ 1691.018456] Key type ._llcrypt unregistered [ 1696.820430] Key type ._llcrypt registered [ 1696.822576] Key type .llcrypt registered [ 1697.020974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1697.039450] alg: No test for adler32 (adler32-zlib) [ 1697.947270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1697.971358] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1704.905993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1710.493345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.145@tcp [ 1710.867412] LNet: 95069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.876045] LNetError: 95069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.893359] LNet: Removed LNI 192.168.202.45@tcp [ 1711.281569] Key type .llcrypt unregistered [ 1711.285262] Key type ._llcrypt unregistered [ 1722.373185] Key type ._llcrypt registered [ 1722.374422] Key type .llcrypt registered [ 1723.408849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1724.877062] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1726.548805] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:47:11 (1785221231) [ 1727.919152] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1729.479178] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:47:14 (1785221234) [ 1731.096752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg245-server [ 1732.738137] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:47:17 (1785221237) [ 1734.496281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg245-server [ 1736.141278] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:47:21 (1785221241) [ 1737.770928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1737.814700] alg: No test for adler32 (adler32-zlib) [ 1738.927941] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 1739.193268] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1740.944501] Key type lgssc registered [ 1742.679799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1762.578303] Key type lgssc unregistered [ 1762.801475] LNet: 98245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.806367] LNetError: 98245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.818723] LNet: Removed LNI 192.168.202.45@tcp [ 1763.415646] Key type .llcrypt unregistered [ 1763.416727] Key type ._llcrypt unregistered [ 1777.323665] Key type ._llcrypt registered [ 1777.326569] Key type .llcrypt registered [ 1778.439412] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:48:03 (1785221283) [ 1778.967575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1778.985335] alg: No test for adler32 (adler32-zlib) [ 1779.898346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.959801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1780.027731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1780.097654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1780.182688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1785.922935] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:48:11 (1785221291) [ 1785.999974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.024147] LNet: 99653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.067568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.135854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.192958] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1787.035548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1787.057264] LNet: 99804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.064896] LNetError: 99804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.080630] LNet: Removed LNI 192.168.202.45@tcp [ 1787.109729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.169827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1787.193854] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1787.824996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1787.848621] LNetError: 99833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.863961] LNet: Removed LNI 192.168.202.45@tcp [ 1787.882963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.942303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1787.966573] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1794.073419] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:48:19 (1785221299) [ 1794.156263] LNet: 100418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.163780] LNet: 100418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1794.170842] LNetError: 100418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.187128] LNet: Removed LNI 192.168.202.45@tcp [ 1794.722662] Key type .llcrypt unregistered [ 1794.724870] Key type ._llcrypt unregistered [ 1799.109193] Key type ._llcrypt registered [ 1799.111887] Key type .llcrypt registered [ 1799.271817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1799.286836] alg: No test for adler32 (adler32-zlib) [ 1800.164589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.228840] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1800.231105] LNet: *** cfs_fail_loc=e003, val=100*** [ 1800.752472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1800.771652] LNet: *** cfs_fail_loc=e003, val=100*** [ 1800.773932] LNet: Skipped 2 previous similar messages [ 1800.777728] Ping reply received with mapping: [ 1800.779233] found metadata reply NI[192.168.202.45@tcp] [ 1800.783351] 2d ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1800.787125] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1800.788839] 5a 5a 5a 5a 5a 5a 5a 5a [ 1800.877166] LNet: 100822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.881427] LNetError: 100822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.895231] LNet: Removed LNI 192.168.202.45@tcp [ 1802.516565] Key type .llcrypt unregistered [ 1802.518831] Key type ._llcrypt unregistered [ 1811.442560] Key type ._llcrypt registered [ 1811.447256] Key type .llcrypt registered [ 1812.571432] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:48:37 (1785221317) [ 1813.157356] Key type .llcrypt unregistered [ 1813.159810] Key type ._llcrypt unregistered [ 1818.897956] Key type ._llcrypt registered [ 1818.899756] Key type .llcrypt registered [ 1819.047579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1819.069908] alg: No test for adler32 (adler32-zlib) [ 1820.304183] LNetError: 101868:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1820.590527] Key type .llcrypt unregistered [ 1820.592960] Key type ._llcrypt unregistered [ 1825.268239] Key type ._llcrypt registered [ 1825.269748] Key type .llcrypt registered [ 1826.309288] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:48:51 (1785221331) [ 1826.766821] Key type .llcrypt unregistered [ 1826.768890] Key type ._llcrypt unregistered [ 1832.523514] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1832.530174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1835.069676] Key type ._llcrypt registered [ 1835.072418] Key type .llcrypt registered [ 1835.250521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1835.264711] alg: No test for adler32 (adler32-zlib) [ 1836.200079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.324797] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1844.622655] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:49:09 (1785221349) [ 1844.753138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.785583] LNet: 103646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.796149] LNetError: 103646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1844.813647] LNet: Removed LNI 10.1.2.3@tcp [ 1844.844339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.483548] Key type .llcrypt unregistered [ 1845.485523] Key type ._llcrypt unregistered [ 1851.146923] Key type ._llcrypt registered [ 1851.148648] Key type .llcrypt registered [ 1851.321596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1851.339040] alg: No test for adler32 (adler32-zlib) [ 1852.242845] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1852.288476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1852.424380] LNet: 103936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.430044] LNetError: 103936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.442165] LNet: Removed LNI 192.168.202.45@tcp [ 1853.018794] Key type .llcrypt unregistered [ 1853.022447] Key type ._llcrypt unregistered [ 1864.683770] Key type ._llcrypt registered [ 1864.685909] Key type .llcrypt registered [ 1866.010441] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:49:30 (1785221370) [ 1866.663311] Key type .llcrypt unregistered [ 1866.667515] Key type ._llcrypt unregistered [ 1872.473098] Key type ._llcrypt registered [ 1872.474748] Key type .llcrypt registered [ 1872.686197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1872.707410] alg: No test for adler32 (adler32-zlib) [ 1879.227188] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:49:44 (1785221384) [ 1879.364026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1879.416033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1879.561395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1879.589613] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1880.319798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1880.368422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1880.409634] LNet: Removed LNI 192.168.202.45@tcp [ 1880.604401] LNet: 105679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.999623] Key type .llcrypt unregistered [ 1881.001220] Key type ._llcrypt unregistered [ 1891.025967] Key type ._llcrypt registered [ 1891.028322] Key type .llcrypt registered [ 1892.290528] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:49:57 (1785221397) [ 1892.773028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1892.790838] alg: No test for adler32 (adler32-zlib) [ 1893.715673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.772325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1893.788444] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1894.518465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1894.559497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.45@tcp [ 1894.575867] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1894.579251] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1894.581572] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1894, 0, 0 [ 1894.585669] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1894.588319] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1894.591490] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1894.594543] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1894, 0, 0 [ 1894.599272] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1894.603546] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1894.605954] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1894.619682] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1894, 0, 1944 [ 1894.626504] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1894.629207] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1894.632640] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1894.639504] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1894, 0, 1944 [ 1894.645643] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1900.960752] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:50:05 (1785221405) [ 1901.405872] LNet: 107219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.414079] LNetError: 107219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.426954] LNet: Removed LNI 192.168.202.45@tcp [ 1902.201442] Key type .llcrypt unregistered [ 1902.206518] Key type ._llcrypt unregistered [ 1907.805355] Key type ._llcrypt registered [ 1907.807875] Key type .llcrypt registered [ 1908.058929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1908.074507] alg: No test for adler32 (adler32-zlib) [ 1909.038533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1909.103053] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1916.594941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 1924.879416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1924.910974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1924.930763] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 1925.687971] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1925.787324] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1925.793083] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1925.797470] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1928.147657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1928.167227] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1936.842311] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:50:41 (1785221441) [ 1936.924655] LNet: 108718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.951851] LNetError: 108718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.958882] LNet: Removed LNI 192.168.202.45@tcp [ 1938.016361] LNet: Removed LNI 192.168.202.45@tcp1 [ 1938.021653] LNetError: 108718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1938.037257] LNetError: 108718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1938.076879] LNet: Removed LNI 10.1.2.3@tcp99 [ 1938.720170] Key type .llcrypt unregistered [ 1938.723054] Key type ._llcrypt unregistered [ 1944.508549] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1944.513512] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1947.030197] Key type ._llcrypt registered [ 1947.032698] Key type .llcrypt registered [ 1947.221702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1947.236403] alg: No test for adler32 (adler32-zlib) [ 1948.139668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.188516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.207103] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1948.793727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1948.810427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1950.475517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1950.529199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1950.565775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1950.579313] LNetError: 109470:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1950.606609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1950.638339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 1950.997891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1951.080906] LNet: 109490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.084992] LNetError: 109490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1951.092134] LNet: Removed LNI 10.1.2.3@tcp [ 1952.103486] LNet: Removed LNI 192.168.202.45@tcp [ 1952.594570] Key type .llcrypt unregistered [ 1952.603052] Key type ._llcrypt unregistered [ 1962.486001] Key type ._llcrypt registered [ 1962.491282] Key type .llcrypt registered [ 1963.618246] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:51:08 (1785221468) [ 1964.141169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.151410] alg: No test for adler32 (adler32-zlib) [ 1965.097450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.149671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.162786] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1965.851524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 1965.941719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg245-client@tcp [ 1971.666715] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:51:16 (1785221476) [ 1971.769331] LNet: 111021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.774524] LNetError: 111021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1971.800049] LNet: Removed LNI 192.168.202.45@tcp [ 1972.332797] Key type .llcrypt unregistered [ 1972.337612] Key type ._llcrypt unregistered [ 1977.548757] Key type ._llcrypt registered [ 1977.550378] Key type .llcrypt registered [ 1977.687661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.702739] alg: No test for adler32 (adler32-zlib) [ 1978.669669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1978.723654] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1978.753595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1984.418898] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:51:29 (1785221489) [ 1985.744841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1987.238278] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:51:32 (1785221492) [ 1988.707337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1989.970853] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:51:35 (1785221495) [ 1990.197942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1990.222996] LNet: 112443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.228435] LNetError: 112443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.239298] LNet: Removed LNI 192.168.202.45@tcp [ 1990.262580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.314386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1990.333560] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 1990.998390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1991.054959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1996.838508] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:51:41 (1785221501) [ 1996.946145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1996.972533] LNet: 113145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1996.978250] LNetError: 113145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1997.990531] LNet: Removed LNI 192.168.202.45@tcp [ 1998.010756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1998.070608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1998.107714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1998.336747] LNet: 113189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.724630] Key type .llcrypt unregistered [ 1998.726750] Key type ._llcrypt unregistered [ 2008.074985] Key type ._llcrypt registered [ 2008.077597] Key type .llcrypt registered [ 2009.023103] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:51:54 (1785221514) [ 2009.327199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2009.342624] alg: No test for adler32 (adler32-zlib) [ 2010.237736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.280135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2010.295507] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2010.849378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2010.904349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2016.457167] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:52:01 (1785221521) [ 2016.599398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2016.625639] LNet: 114706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.635788] LNetError: 114706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.654920] LNet: Removed LNI 192.168.202.45@tcp [ 2016.679186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.717825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2016.744774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2021.909911] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:52:07 (1785221527) [ 2022.099893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2022.116578] LNet: 115288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.136258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.180628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2022.489912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2022.538752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2022.611462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2023.026830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2023.050476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2023.087409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2023.106787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.143780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2023.459946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2023.492799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2023.554969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2023.914384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2023.954534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2028.911654] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:52:14 (1785221534) [ 2029.058224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2029.073981] LNet: 115921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.076744] LNet: 115921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2029.093204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.124298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2029.142200] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2029.657553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2029.671490] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2030.487512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2030.531709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2030.655897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2030.673547] LNetError: 116200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.677934] LNet: Removed LNI 192.168.202.45@tcp [ 2030.687382] LNet: Removed LNI 192.168.202.45@tcp2 [ 2030.705227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.756601] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2030.762125] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2030.768702] LNet: 116206:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2031.135492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2031.283741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.302289] LNet: 116243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.305530] LNet: 116243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2032.351234] LNet: Removed LNI 192.168.202.45@tcp [ 2032.354069] LNetError: 116243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.360700] LNetError: 116243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2032.380374] LNet: Removed LNI 192.168.202.45@tcp2 [ 2032.412579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.462478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2032.485398] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2033.150459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2033.171333] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2033.866886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2033.905048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2033.950924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2034.089602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.167134] LNet: Removed LNI 192.168.202.45@tcp [ 2035.168987] LNetError: 116527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.186068] LNetError: 116527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2035.203889] LNet: Removed LNI 192.168.202.45@tcp2 [ 2035.244670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.326553] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2035.335899] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2035.344496] LNet: 116534:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2035.685977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2035.829587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.846803] LNet: 116571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.850239] LNet: 116571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2035.855156] LNet: Removed LNI 192.168.202.45@tcp [ 2036.904509] LNet: Removed LNI 192.168.202.45@tcp2 [ 2036.930172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.995572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2037.014216] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2037.555944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2037.571258] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2038.227405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2038.267743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2038.512915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2039.583637] LNet: Removed LNI 192.168.202.45@tcp [ 2039.588171] LNetError: 116850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2039.594298] LNetError: 116850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2039.615980] LNet: Removed LNI 192.168.202.45@tcp2 [ 2039.640966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.683283] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2039.687515] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2039.691832] LNet: 116856:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2040.024862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2040.217523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2041.247627] LNet: Removed LNI 192.168.202.45@tcp [ 2041.269715] LNet: Removed LNI 192.168.202.45@tcp2 [ 2041.317736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.374427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2041.395965] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2041.905838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2041.923844] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2042.612814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2042.647791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2042.692598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2042.848918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2043.873460] LNet: Removed LNI 192.168.202.45@tcp [ 2044.905918] LNet: Removed LNI 192.168.202.45@tcp2 [ 2044.940771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.015408] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2045.019952] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2045.024359] LNet: 117184:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2045.444575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2045.598650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2045.620852] LNet: 117221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.625493] LNet: 117221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2045.630072] LNet: Removed LNI 192.168.202.45@tcp [ 2046.700486] LNet: Removed LNI 192.168.202.45@tcp2 [ 2046.737856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.802429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2046.829239] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2047.406985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2047.420393] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2047.951908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2047.977834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2048.028416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2048.348704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2048.365525] LNetError: 117517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.370671] LNetError: 117517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2048.378680] LNet: Removed LNI 192.168.202.45@tcp [ 2048.386982] LNet: Removed LNI 192.168.202.45@tcp2 [ 2048.407077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2048.463928] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2048.468678] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2048.896462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2049.141646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2050.207359] LNet: Removed LNI 192.168.202.45@tcp [ 2050.223945] LNet: Removed LNI 192.168.202.45@tcp2 [ 2050.261629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.299510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2050.322709] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2050.875039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2050.914452] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2051.603121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2051.639763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2051.679601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2051.738304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2052.014504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.087162] LNet: Removed LNI 192.168.202.45@tcp [ 2054.122614] LNet: Removed LNI 192.168.202.45@tcp2 [ 2054.147623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.203306] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2054.209272] LNet: Added LNI 192.168.202.45@tcp2 [8/256/0/180] [ 2054.215337] LNet: 117883:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2054.228533] LNet: 117883:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 2054.748450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2060.483947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2061.784917] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:52:47 (1785221567) [ 2061.933616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.968105] LNet: 118638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.975205] LNet: 118638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2061.982374] LNet: Removed LNI 192.168.202.45@tcp [ 2063.015761] LNet: Removed LNI 192.168.202.45@tcp2 [ 2063.056708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.335979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2063.363956] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2063.966134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2063.986420] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2064.652335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2064.679955] LNetError: 118916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2064.686410] LNetError: 118916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2064.691727] LNet: Removed LNI 10.1.2.3@tcp [ 2064.698716] LNet: Removed LNI 192.168.202.45@tcp [ 2064.716315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.775248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2064.793964] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2064.796977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2065.259423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2065.290716] LNet: Removed LNI 192.168.202.45@tcp [ 2065.307355] LNet: Removed LNI 10.1.2.3@tcp [ 2070.690662] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 02:52:55 (1785221575) [ 2070.792423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2070.826626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.888144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2070.910651] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2071.463648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2071.494480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 2071.522849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 2071.550649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 2071.574667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 2071.597228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 2071.619633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 2071.652233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 2071.676385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 2071.704285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 2071.728599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.45@tcp [ 2071.900909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2077.222299] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:53:02 (1785221582) [ 2077.327417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.450435] LNet: Removed LNI 192.168.202.45@tcp [ 2078.493147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.605332] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2078.783052] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2078.792246] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2078.798589] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2081.068877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2081.085880] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2081.700571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2081.716776] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2083.357732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.45@tcp [ 2083.394224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2083.454445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.45@tcp -m PUT -e local_timeout -r 1 [ 2083.531203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2083.604302] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2083.607259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2083.772158] LNetError: 120321:0:(lib-move.c:4801:lnet_parse()) 192.168.202.45@tcp, src 192.168.202.45@tcp: Dropping PUT (error -108 looking up sender) [ 2085.855169] LNet: Removed LNI 10.1.2.3@tcp [ 2086.888975] LNet: Removed LNI 192.168.202.45@tcp [ 2087.171489] Key type .llcrypt unregistered [ 2087.172916] Key type ._llcrypt unregistered [ 2097.747429] Key type ._llcrypt registered [ 2097.749707] Key type .llcrypt registered [ 2098.799911] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:53:23 (1785221603) [ 2099.191418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.202667] alg: No test for adler32 (adler32-zlib) [ 2100.138384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.493801] LNet: 121631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.179366] Key type .llcrypt unregistered [ 2101.181058] Key type ._llcrypt unregistered [ 2106.838994] Key type ._llcrypt registered [ 2106.843627] Key type .llcrypt registered [ 2107.010736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2107.024687] alg: No test for adler32 (adler32-zlib) [ 2107.899585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2107.923289] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2114.223650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 2114.469934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2115.096401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.145@tcp [ 2115.326123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2115.352965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2115.378725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2115.399321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2115.440586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2115.476502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2115.504308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2115.548521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2115.577583] LNet: 122081:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2115.609570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2115.652780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2115.703852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2115.742422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2115.769669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2115.799498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2116.494223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2116.736766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2116.781125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2116.815325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2116.851151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2116.873394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2116.899958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2116.942671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2116.971035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2116.982914] LNet: 122199:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2117.000328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2117.028877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2117.053322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2117.081263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2117.117672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2117.156596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2117.669713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.145@tcp [ 2117.900199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2117.929723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2117.967866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2117.995851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2118.023916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2118.048941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2118.091073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2118.123153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2118.150561] LNet: 122316:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2118.171738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2118.188177] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.145@tcp [ 2118.202934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2118.230552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2118.262681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2118.287643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2118.318136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2118.924474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2119.145584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2119.172337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2119.207360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2119.263411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2119.310766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2119.343488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2119.375904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2119.410391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2119.437584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2119.461626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2119.490877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2119.524149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2119.559881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2119.590272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2119.967550] LNet: 122479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.975715] LNetError: 122479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.989281] LNet: Removed LNI 192.168.202.45@tcp [ 2120.419935] Key type .llcrypt unregistered [ 2120.429273] Key type ._llcrypt unregistered [ 2132.735657] Key type ._llcrypt registered [ 2132.737517] Key type .llcrypt registered [ 2133.678451] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:53:58 (1785221638) [ 2133.895015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2133.905130] alg: No test for adler32 (adler32-zlib) [ 2134.752719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.013732] LNet: 123366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.464251] Key type .llcrypt unregistered [ 2135.466849] Key type ._llcrypt unregistered [ 2140.438904] Key type ._llcrypt registered [ 2140.441077] Key type .llcrypt registered [ 2140.641285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2140.664145] alg: No test for adler32 (adler32-zlib) [ 2141.596519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2141.616106] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2147.224584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.145@tcp [ 2153.098425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.128761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2153.145116] LNet: Added LNI 192.168.202.45@tcp1 [8/256/0/180] [ 2153.819572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2154.204850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.145@tcp [ 2154.415633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2154.438984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2154.462822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2154.485943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2154.512890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2154.544588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2154.577980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2154.604136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2154.621893] LNet: 124018:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2154.635936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2154.657288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2154.691627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2154.727777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2154.758916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2154.785732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2155.309159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2155.516852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2155.546983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2155.570681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2155.597940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2155.623704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2155.653029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2155.679903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2155.707070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2155.719351] LNet: 124135:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2155.741133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2155.770810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2155.804207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2155.829881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2155.860239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2155.890596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2156.407849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.145@tcp [ 2156.586720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2156.607783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2156.630688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2156.654817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2156.680770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2156.704212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2156.733417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2156.768251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2156.793661] LNet: 124252:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2156.818316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2156.845806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2156.878596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2156.914757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2156.942754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2156.974114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2157.436793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2157.646399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2157.666930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2157.695594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2157.714295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2157.737927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2157.759879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2157.784508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2157.807893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2157.832423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2157.861716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2157.895365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2157.916517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2157.945443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2157.968819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2161.986149] LNet: 124450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.997731] LNetError: 124450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.002728] LNet: Removed LNI 192.168.202.45@tcp [ 2162.019697] LNet: Removed LNI 192.168.202.45@tcp1 [ 2162.337689] Key type .llcrypt unregistered [ 2162.343481] Key type ._llcrypt unregistered [ 2174.486142] Key type ._llcrypt registered [ 2174.488398] Key type .llcrypt registered [ 2175.398463] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 02:54:40 (1785221680) [ 2176.714290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2178.034326] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:54:43 (1785221683) [ 2183.537782] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:54:48 (1785221688) [ 2188.998744] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:54:54 (1785221694) [ 2194.287869] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:54:59 (1785221699) [ 2199.691667] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:55:05 (1785221705) [ 2204.954169] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:55:10 (1785221710) [ 2209.967612] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:55:15 (1785221715) [ 2214.697508] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:55:20 (1785221720) [ 2219.386285] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:55:24 (1785221724) [ 2223.341985] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:55:28 (1785221728) [ 2227.845933] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:55:33 (1785221733) [ 2233.211058] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:55:38 (1785221738) [ 2238.940330] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:55:44 (1785221744) [ 2244.288352] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:55:49 (1785221749) [ 2249.630427] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:55:54 (1785221754) [ 2254.655786] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:56:00 (1785221760) [ 2261.110518] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:56:06 (1785221766) [ 2266.562930] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:56:11 (1785221771) [ 2266.851795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2266.869183] alg: No test for adler32 (adler32-zlib) [ 2267.746396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.899539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.920332] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2268.919924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.933663] LNet: 135445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.940959] LNetError: 135445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.950918] LNet: Removed LNI 192.168.202.45@tcp [ 2268.968948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.014210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2269.030865] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2273.910705] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:56:19 (1785221779) [ 2279.827787] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:56:24 (1785221784) [ 2285.401082] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:56:30 (1785221790) [ 2290.782594] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:56:36 (1785221796) [ 2295.867840] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:56:41 (1785221801) [ 2301.407909] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:56:46 (1785221806) [ 2307.022944] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:56:52 (1785221812) [ 2311.700960] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:56:57 (1785221817) [ 2316.346688] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:57:01 (1785221821) [ 2320.975949] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:57:06 (1785221826) [ 2325.232251] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:57:10 (1785221830) [ 2329.680701] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:57:15 (1785221835) [ 2333.814705] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2067 sec ========== 02:57:19 (1785221839) [ 2333.986673] LNet: 142908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.989414] LNetError: 142908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.001856] LNet: Removed LNI 192.168.202.45@tcp [ 2334.399416] Key type .llcrypt unregistered [ 2334.401565] Key type ._llcrypt unregistered [ 2342.018723] Key type ._llcrypt registered [ 2342.020581] Key type .llcrypt registered [ 2342.281127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2342.291793] alg: No test for adler32 (adler32-zlib) [ 2343.208126] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 2343.331841] LNet: Added LNI 192.168.202.45@tcp [8/256/0/180] [ 2344.927237] Key type lgssc registered [ 2345.653400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2383.109590] Lustre: Mounted lustre-client [ 2386.536401] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2390.621407] Lustre: Unmounted lustre-client [ 2402.541451] Key type lgssc unregistered [ 2402.750522] LNet: 145828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2402.757412] LNetError: 145828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2402.778086] LNet: Removed LNI 192.168.202.45@tcp [ 2403.248275] Key type .llcrypt unregistered [ 2403.251842] Key type ._llcrypt unregistered