[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473112469 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001024] APIC: Switch to symmetric I/O mode setup [ 0.003344] x2apic enabled [ 0.004019] Switched APIC routing to physical x2apic. [ 0.005025] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009049] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010035] pid_max: default: 32768 minimum: 301 [ 0.011250] LSM: Security Framework initializing [ 0.012081] Yama: becoming mindful. [ 0.013071] SELinux: Initializing. [ 0.015127] *** VALIDATE selinux *** [ 0.029043] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.035331] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.036288] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037221] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.038194] *** VALIDATE tmpfs *** [ 0.039619] *** VALIDATE proc *** [ 0.041350] *** VALIDATE cgroup *** [ 0.042027] *** VALIDATE cgroup2 *** [ 0.043431] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.045135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.046022] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.047049] Spectre V2 : User space: Vulnerable [ 0.048012] Speculative Store Bypass: Vulnerable [ 0.051425] debug: unmapping init [mem 0xffffffff87259000-0xffffffff87260fff] [ 0.054364] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.056000] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.057047] ... version: 2 [ 0.058014] ... bit width: 48 [ 0.059010] ... generic registers: 4 [ 0.060019] ... value mask: 0000ffffffffffff [ 0.061024] ... max period: 00007fffffffffff [ 0.062015] ... fixed-purpose events: 3 [ 0.063017] ... event mask: 000000070000000f [ 0.064540] rcu: Hierarchical SRCU implementation. [ 0.066823] smp: Bringing up secondary CPUs ... [ 0.067841] x86: Booting SMP configuration: [ 0.068041] .... node #0, CPUs: #1 #2 #3 [ 0.078750] smp: Brought up 1 node, 4 CPUs [ 0.080051] smpboot: Max logical packages: 1 [ 0.081021] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.120058] node 0 deferred pages initialised in 37ms [ 0.124037] devtmpfs: initialized [ 0.125210] x86/mm: Memory block size: 128MB [ 0.129455] gcov: version magic: 0x41383552 [ 0.132376] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135143] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138584] pinctrl core: initialized pinctrl subsystem [ 0.140387] [ 0.140932] ************************************************************* [ 0.144015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146011] ** ** [ 0.149013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151011] ** ** [ 0.154019] ** This means that this kernel is built to expose internal ** [ 0.156012] ** IOMMU data structures, which may compromise security on ** [ 0.158012] ** your system. ** [ 0.159010] ** ** [ 0.161010] ** If you see this message and you are not debugging the ** [ 0.162009] ** kernel, report this immediately to your vendor! ** [ 0.164010] ** ** [ 0.166016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168016] ************************************************************* [ 0.170785] NET: Registered protocol family 16 [ 0.172604] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175208] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182140] cpuidle: using governor menu [ 0.183779] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184432] PCI: Using configuration type 1 for base access [ 0.185131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192433] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195085] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197265] cryptd: max_cpu_qlen set to 1000 [ 0.199198] ACPI: Added _OSI(Module Device) [ 0.201017] ACPI: Added _OSI(Processor Device) [ 0.203015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208609] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214329] ACPI: Interpreter enabled [ 0.215000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218011] ACPI: Using IOAPIC for interrupt routing [ 0.221512] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228836] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246244] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.251019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258409] acpiphp: Slot [2] registered [ 0.260134] acpiphp: Slot [5] registered [ 0.261239] acpiphp: Slot [6] registered [ 0.263110] acpiphp: Slot [3] registered [ 0.264168] acpiphp: Slot [4] registered [ 0.265107] acpiphp: Slot [7] registered [ 0.267238] acpiphp: Slot [8] registered [ 0.268219] acpiphp: Slot [9] registered [ 0.270151] acpiphp: Slot [10] registered [ 0.272228] acpiphp: Slot [11] registered [ 0.273160] acpiphp: Slot [12] registered [ 0.275563] acpiphp: Slot [13] registered [ 0.277247] acpiphp: Slot [14] registered [ 0.279210] acpiphp: Slot [15] registered [ 0.281149] acpiphp: Slot [16] registered [ 0.282171] acpiphp: Slot [17] registered [ 0.284316] acpiphp: Slot [18] registered [ 0.286243] acpiphp: Slot [19] registered [ 0.287218] acpiphp: Slot [20] registered [ 0.288210] acpiphp: Slot [21] registered [ 0.290157] acpiphp: Slot [22] registered [ 0.291201] acpiphp: Slot [23] registered [ 0.294237] acpiphp: Slot [24] registered [ 0.297131] acpiphp: Slot [25] registered [ 0.299205] acpiphp: Slot [26] registered [ 0.300222] acpiphp: Slot [27] registered [ 0.302279] acpiphp: Slot [28] registered [ 0.304130] acpiphp: Slot [29] registered [ 0.305167] acpiphp: Slot [30] registered [ 0.306096] acpiphp: Slot [31] registered [ 0.307092] PCI host bridge to bus 0000:00 [ 0.309020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.312023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.314037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.320024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.322039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.327016] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.331464] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341000] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.345074] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357825] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.360798] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.361101] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365308] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372022] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.385017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.391016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.401277] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.428030] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.442369] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.464021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.490022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.504902] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.508415] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.510361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.513379] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.515253] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.521028] iommu: Default domain type: Passthrough [ 0.523546] SCSI subsystem initialized [ 0.524144] ACPI: bus type USB registered [ 0.526108] usbcore: registered new interface driver usbfs [ 0.528130] usbcore: registered new interface driver hub [ 0.532123] usbcore: registered new device driver usb [ 0.536359] pps_core: LinuxPPS API ver. 1 registered [ 0.538009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.541210] PTP clock support registered [ 0.545114] EDAC MC: Ver: 3.0.0 [ 0.549213] PCI: Using ACPI for IRQ routing [ 0.551783] NetLabel: Initializing [ 0.553015] NetLabel: domain hash size = 128 [ 0.555010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.557119] NetLabel: unlabeled traffic allowed by default [ 0.560047] vgaarb: loaded [ 0.561354] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.564021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.571000] clocksource: Switched to clocksource kvm-clock [ 0.744277] VFS: Disk quotas dquot_6.6.0 [ 0.751353] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.759448] *** VALIDATE ramfs *** [ 0.760521] *** VALIDATE hugetlbfs *** [ 0.766495] pnp: PnP ACPI init [ 0.773295] pnp: PnP ACPI: found 6 devices [ 0.802460] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.812844] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.816350] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.819809] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.823061] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.826719] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.830636] NET: Registered protocol family 2 [ 0.833417] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.839644] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.850864] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.861569] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.867360] TCP: Hash tables configured (established 65536 bind 65536) [ 0.872127] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.878827] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.885055] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.891139] NET: Registered protocol family 1 [ 0.900614] RPC: Registered named UNIX socket transport module. [ 0.902312] RPC: Registered udp transport module. [ 0.905610] RPC: Registered tcp transport module. [ 0.908167] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.911457] NET: Registered protocol family 44 [ 0.913508] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.916348] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.920343] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.923701] PCI: CLS 0 bytes, default 64 [ 0.927925] Unpacking initramfs... [ 3.275591] debug: unmapping init [mem 0xffff96b13cc64000-0xffff96b13ffcffff] [ 3.283421] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.288703] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.295559] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.081572] Initialise system trusted keyrings [ 4.086920] Key type blacklist registered [ 4.095205] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.108964] zbud: loaded [ 4.112833] *** VALIDATE nfs *** [ 4.114296] *** VALIDATE nfs4 *** [ 4.116334] pstore: using deflate compression [ 4.121936] Platform Keyring initialized [ 4.349805] NET: Registered protocol family 38 [ 4.351755] Key type asymmetric registered [ 4.353580] Asymmetric key parser 'x509' registered [ 4.356847] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.362287] io scheduler mq-deadline registered [ 4.363994] io scheduler kyber registered [ 4.366586] io scheduler bfq registered [ 4.368667] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.371762] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.375460] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.380035] ACPI: Power Button [PWRF] [ 4.389437] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.401580] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.417746] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.446841] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.477293] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.485417] Non-volatile memory driver v1.3 [ 4.486939] Linux agpgart interface v0.103 [ 4.540361] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 4.543794] vda: detected capacity change from 0 to 74711040 [ 4.563898] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.569564] vdb: detected capacity change from 0 to 1073741824 [ 4.586704] libphy: Fixed MDIO Bus: probed [ 4.599729] usbcore: registered new interface driver usbserial_generic [ 4.603256] usbserial: USB Serial support registered for generic [ 4.606892] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.629949] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.636648] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.645888] mousedev: PS/2 mouse device common for all mice [ 4.663276] rtc_cmos 00:05: RTC can wake from S4 [ 4.678486] rtc_cmos 00:05: registered as rtc0 [ 4.682855] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.684485] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.691883] intel_pstate: CPU model not supported [ 4.699175] hid: raw HID events driver (C) Jiri Kosina [ 4.709791] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.716897] usbcore: registered new interface driver usbhid [ 4.723827] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.732022] usbhid: USB HID core driver [ 4.739995] drop_monitor: Initializing network drop monitor service [ 4.750187] Initializing XFRM netlink socket [ 4.758389] NET: Registered protocol family 10 [ 4.764631] Segment Routing with IPv6 [ 4.767755] NET: Registered protocol family 17 [ 4.772776] mpls_gso: MPLS GSO support [ 4.788587] RAS: Correctable Errors collector initialized. [ 4.791777] AVX version of gcm_enc/dec engaged. [ 4.795174] AES CTR mode by8 optimization enabled [ 4.934219] sched_clock: Marking stable (4934090471, 0)->(5996419045, -1062328574) [ 4.942435] registered taskstats version 1 [ 4.945267] Loading compiled-in X.509 certificates [ 4.949703] zswap: loaded using pool lzo/zbud [ 4.997481] Key type big_key registered [ 5.012140] Key type encrypted registered [ 5.017235] ima: No TPM chip found, activating TPM-bypass! [ 5.019384] ima: Allocated hash algorithm: sha1 [ 5.021071] ima: No architecture policies found [ 5.022543] evm: Initialising EVM extended attributes: [ 5.025266] evm: security.selinux [ 5.026217] evm: security.ima [ 5.027068] evm: security.capability [ 5.028135] evm: HMAC attrs: 0x1 [ 5.030655] rtc_cmos 00:05: setting system clock to 2026-08-17 06:09:23 UTC (1786946963) [ 5.039907] debug: unmapping init [mem 0xffffffff88203000-0xffffffff883fffff] [ 5.042905] debug: unmapping init [mem 0xffffffff86f82000-0xffffffff87258fff] [ 5.053385] Write protecting the kernel read-only data: 28672k [ 5.071230] debug: unmapping init [mem 0xffffffff85603000-0xffffffff857fffff] [ 5.079782] debug: unmapping init [mem 0xffffffff85f14000-0xffffffff85ffffff] [ 5.158525] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.169812] systemd[1]: Detected virtualization kvm. [ 5.172149] systemd[1]: Detected architecture x86-64. [ 5.174219] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.216685] systemd[1]: No hostname configured. [ 5.221197] systemd[1]: Set hostname to . [ 5.226337] random: systemd: uninitialized urandom read (16 bytes read) [ 5.232354] systemd[1]: Initializing machine ID from random generator. [ 5.514244] random: systemd: uninitialized urandom read (16 bytes read) [ 5.520254] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.534651] random: systemd: uninitialized urandom read (16 bytes read) [ 5.543564] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.559626] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ 7.834319] hrtimer: interrupt took 3560723 ns [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.467473] device-mapper: uevent: version 1.0.3 [ 8.475603] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.235489] virtio_net virtio0 ens2: renamed from eth0 [ 12.742187] scsi host0: ata_piix [ 12.905972] scsi host1: ata_piix [ 12.916104] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 12.924834] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 17.626371] random: crng init done [ 17.639379] random: 7 urandom warning(s) missed due to ratelimiting [ 21.918992] dracut-initqueue[601]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 24.540538] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 29.074569] printk: systemd: 26 output lines suppressed due to ratelimiting [ 29.843657] SELinux: Disabled at runtime. [ 29.946488] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.965740] systemd[1]: Detected virtualization kvm. [ 29.969714] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 32.521663] systemd[1]: initrd-switch-root.service: Succeeded. [ 32.536521] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 32.548590] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 32.565036] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 32.577481] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 32.596842] systemd[1]: Starting Journal Service... Starting Journal Service... [ 32.630296] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ 33.181838] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 34.793965] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 36.372421] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 36.909151] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 37.943465] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.383561] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit)[ 46.687940] Key type dns_resolver registered [** ] A start job is running for Configur…only root support (15s / no limit) [* ] A start job is running for Configur…only root support (15s / no limit)[ 47.974894] NFS: Registering the id_resolver key type [ 47.981508] Key type id_resolver registered [ 47.988150] Key type id_legacy registered [** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-client login: [ 146.467323] libcfs: loading out-of-tree module taints kernel. [ 146.579512] Key type ._llcrypt registered [ 146.582104] Key type .llcrypt registered [ 147.412194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 147.431046] alg: No test for adler32 (adler32-zlib) [ 149.220924] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 150.244355] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 152.111959] Key type lgssc registered [ 154.241347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 303.076506] Lustre: Mounted lustre-client - version 2.17.57_1_g72a0e6d [ 308.458835] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 324.824978] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing check_logdir /tmp/testlogs/ [ 328.675849] Lustre: lustre-OST0000-osc-ffff96b1829e9800: disconnect after 23s idle [ 330.067691] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing yml_node [ 336.983995] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 339.912177] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 343.355838] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 345.667873] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 17 02:15:02 EDT 2026 [ 366.010779] Lustre: Unmounted lustre-client [ 400.691217] Key type lgssc unregistered [ 401.119322] LNet: 8446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.126703] LNetError: 8446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 401.149812] LNet: Removed LNI 192.168.204.42@tcp [ 402.169697] Key type .llcrypt unregistered [ 402.171850] Key type ._llcrypt unregistered [ 435.526469] Key type ._llcrypt registered [ 435.538713] Key type .llcrypt registered [ 435.952718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 435.963952] alg: No test for adler32 (adler32-zlib) [ 436.903556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 436.979569] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 437.019874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 442.361854] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 442.517040] LNet: 9329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.530282] LNetError: 9329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 442.588196] LNet: Removed LNI 192.168.204.42@tcp [ 443.720616] Key type .llcrypt unregistered [ 443.725755] Key type ._llcrypt unregistered [ 452.377655] Key type ._llcrypt registered [ 452.381576] Key type .llcrypt registered [ 454.250267] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 02:16:50 (1786947410) [ 455.029036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 455.081237] alg: No test for adler32 (adler32-zlib) [ 456.078208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.208930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.229312] LNet: 9752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.317772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.418700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.471987] LNet: 9759:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 468.336204] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 02:17:05 (1786947425) [ 468.454225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.488389] LNet: 10334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.535550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.732556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 478.831251] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 02:17:15 (1786947435) [ 479.108909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.185869] LNet: 10925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 479.225712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.343377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 489.565073] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 02:17:25 (1786947445) [ 489.806273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.858184] LNet: 11507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.910187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.040462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 499.943339] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 02:17:36 (1786947456) [ 500.097608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.142119] LNet: 12089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.178098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.332063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 501.202368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 512.629506] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 02:17:48 (1786947468) [ 513.190913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.239650] LNet: 12687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.265217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.400679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 514.428318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 515.313966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 525.654490] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 02:18:02 (1786947482) [ 526.250791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.338946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.526232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 537.114972] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 02:18:12 (1786947492) [ 537.238980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.279823] LNet: 13862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.288214] LNet: 13862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 537.311747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.445729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 537.485709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 538.195310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 538.245840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 539.134484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 539.255649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 540.372481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 540.495403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 541.602918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 541.655589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 542.795862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 542.853467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 543.574920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 543.651779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 544.530782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 544.584479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 545.047472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 545.078105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 545.653907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 545.719863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 546.297079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 546.394891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 547.144380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 547.201776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 547.963451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 548.029676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 558.462201] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 02:18:34 (1786947514) [ 558.791969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.879854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.009809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 559.073269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 568.397260] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 02:18:44 (1786947524) [ 568.539356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.604160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.737251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 568.770408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 578.640671] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 02:18:55 (1786947535) [ 578.970918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.014861] LNet: 15726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.025973] LNet: 15726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 579.120696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.283365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 579.344250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 589.042361] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 02:19:05 (1786947545) [ 589.357746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.409431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.514605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 589.589382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 598.747602] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 02:19:15 (1786947555) [ 598.989450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.074471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.216317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 599.261594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 608.615957] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 02:19:25 (1786947565) [ 608.941346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.008596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.123613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 609.196577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 618.795460] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 02:19:35 (1786947575) [ 618.945601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.012910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.160818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 619.842128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 628.964184] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 02:19:45 (1786947585) [ 629.330332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.376785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.729785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.141923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 639.734203] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 02:19:56 (1786947596) [ 639.994642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.055703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.258173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 640.538492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 650.147517] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 02:20:06 (1786947606) [ 650.437549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.489845] LNet: 19903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.505833] LNet: 19903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 650.551545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.449504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 652.047671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 660.019291] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 02:20:16 (1786947616) [ 660.208680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.263225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.411307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 660.476976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 668.495854] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 02:20:25 (1786947625) [ 668.719338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.790125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.885547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 668.985503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 677.527603] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 02:20:34 (1786947634) [ 677.650171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.699780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.798797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 686.433816] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 02:20:42 (1786947642) [ 686.616587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.688361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.867907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 696.496394] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 02:20:53 (1786947653) [ 696.620890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.670712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.827447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 697.636849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 706.301607] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 02:21:02 (1786947662) [ 706.458537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.548126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.707354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 706.786530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 715.742304] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 02:21:12 (1786947672) [ 716.086376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.135790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.212140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 716.290704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 725.740494] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 02:21:22 (1786947682) [ 726.149442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.223422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.432798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 726.526650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 735.073551] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 02:21:31 (1786947691) [ 735.222749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.289164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.441630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 735.518261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 744.244401] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 02:21:41 (1786947701) [ 744.691775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.726778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.775739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 744.864692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 744.896497] LNetError: 25883:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 744.971678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 745.018213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 752.493342] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 02:21:49 (1786947709) [ 752.602065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.697108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.872792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.rbtdJbTzYB [ 753.011931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.rbtdJbTzYB [ 753.091542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.rbtdJbTzYB [ 761.182868] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 02:21:58 (1786947718) [ 761.385748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.477278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.620068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 761.673613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 761.750156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 769.937822] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 02:22:06 (1786947726) [ 770.236285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.300673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.495688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 770.565684] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 772.887988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 774.501696] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 776.560236] Key type lgssc registered [ 777.921617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 782.278398] LNetError: 27645:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.42@tcp99 has no available nets [ 785.533435] Key type lgssc unregistered [ 785.919879] LNet: 28755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.924089] LNet: 28755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 785.930447] LNetError: 28755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.948579] LNet: Removed LNI 192.168.204.42@tcp [ 786.772714] Key type .llcrypt unregistered [ 786.775611] Key type ._llcrypt unregistered [ 794.904856] Key type ._llcrypt registered [ 794.906767] Key type .llcrypt registered [ 796.305344] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 02:22:33 (1786947753) [ 796.706683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.718519] alg: No test for adler32 (adler32-zlib) [ 797.663380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.730724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 801.629319] LNet: Message forwarding will be enabled in 2s [ 804.703356] LNet: Message forwarding enabled [ 805.187672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 805.226962] LNet: Message forwarding will stop in 2s to 6s [ 809.932827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 810.564756] LNet: Message forwarding disabled [ 812.231310] LNet: Message forwarding will be enabled in 2s [ 815.712422] LNet: Message forwarding enabled [ 821.811399] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 02:22:58 (1786947778) [ 821.887430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.942670] LNet: 30155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.194897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.321778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 832.689365] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 02:23:09 (1786947789) [ 832.838928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.888360] LNet: 30738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.954420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.020678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 833.050134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 833.078633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 833.108398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 833.147307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 833.249378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 833.321276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 833.385906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 833.444907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 833.514723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 833.579348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 833.639360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 833.721172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 833.768399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 833.831562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 833.894967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 833.986703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 834.082864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 834.216264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 834.292733] LNetError: 30788:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 843.447510] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 02:23:19 (1786947799) [ 843.571132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.653736] LNet: 31355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.695715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.865635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 852.907633] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 02:23:29 (1786947809) [ 855.420571] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg442-server [ 857.429250] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 02:23:34 (1786947814) [ 857.776312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.806738] LNet: 32291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.832448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.920416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.958835] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 859.110959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[43-47]@tcp [ 859.152245] LNet: 32424:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 859.265263] LNetError: 32295:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.46@tcp is being used as a gateway but routing feature is not turned on [ 859.298991] LNetError: 32295:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.46@tcp has gone from up to down [ 860.212093] LNetError: 32295:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 860.238187] LNetError: 32295:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 861.222077] LNetError: 32295:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.47@tcp is being used as a gateway but routing feature is not turned on [ 861.245286] LNetError: 32295:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 863.268239] LNetError: 32295:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.43@tcp is being used as a gateway but routing feature is not turned on [ 863.294416] LNetError: 32295:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 867.382217] LNetError: 32295:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.43@tcp is being used as a gateway but routing feature is not turned on [ 867.403639] LNetError: 32295:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 867.678330] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 02:23:44 (1786947824) [ 869.830354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg442-server [ 871.836566] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 02:23:48 (1786947828) [ 872.334360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.383578] LNet: 33361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.397837] LNetError: 33361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.435417] LNet: Removed LNI 192.168.204.42@tcp [ 872.587654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.762685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.822617] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 873.669358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[43-48/2]@tcp [ 873.682101] LNet: 33478:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 873.686928] LNet: 33478:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 873.715200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[43-48/2]@tcp [ 873.720318] LNetError: 33365:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.47@tcp has gone from up to down [ 873.728260] LNetError: 33365:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 881.784775] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 02:23:58 (1786947838) [ 882.028509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.074920] LNet: 34069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.087110] LNetError: 34069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.133992] LNet: Removed LNI 192.168.204.42@tcp [ 882.169962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.295418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 882.353106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.410565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.502862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 882.606884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.661540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.804926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 882.969506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.045568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.144058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 883.295985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.354595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.432212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 883.554314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.603081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.660609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 883.752908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.813343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.965165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 884.089453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.160134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.254316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 884.319436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.344680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.394960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 884.466245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.522914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.666974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 884.777679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.873698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.023924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 885.127874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.157721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.222996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 893.906261] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 02:24:10 (1786947850) [ 896.940781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg442-server [ 899.692174] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 02:24:16 (1786947856) [ 902.353318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg442-server [ 904.536417] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 02:24:21 (1786947861) [ 904.695220] LNet: 35465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.705609] LNet: 35465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 905.488809] Key type .llcrypt unregistered [ 905.491371] Key type ._llcrypt unregistered [ 913.320426] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 913.586550] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 913.596960] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 913.606186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 916.295246] Key type ._llcrypt registered [ 916.299194] Key type .llcrypt registered [ 916.535305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 916.570195] alg: No test for adler32 (adler32-zlib) [ 917.598547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.696176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 917.765463] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 918.436495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 918.456679] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 920.878649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 920.897392] LNet: Removed LNI 10.1.2.3@tcp [ 920.987267] LNet: 36201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.993897] LNetError: 36201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.005762] LNet: Removed LNI 192.168.204.42@tcp [ 921.856690] Key type .llcrypt unregistered [ 921.859343] Key type ._llcrypt unregistered [ 936.938611] Key type ._llcrypt registered [ 936.948680] Key type .llcrypt registered [ 938.557224] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 02:24:55 (1786947895) [ 939.528735] Key type .llcrypt unregistered [ 939.535311] Key type ._llcrypt unregistered [ 948.391448] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 948.540185] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 948.548083] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 948.558407] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.571633] Key type ._llcrypt registered [ 951.577654] Key type .llcrypt registered [ 951.819388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 951.844178] alg: No test for adler32 (adler32-zlib) [ 952.808518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.873389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 952.913204] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 953.944989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 953.984752] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 957.468570] LNet: 37704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.475541] LNetError: 37704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 957.481337] LNet: Removed LNI 10.1.2.3@tcp [ 957.490981] LNet: Removed LNI 192.168.204.42@tcp [ 958.409979] Key type .llcrypt unregistered [ 958.412879] Key type ._llcrypt unregistered [ 975.040624] Key type ._llcrypt registered [ 975.043327] Key type .llcrypt registered [ 976.726770] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 02:25:33 (1786947933) [ 977.441574] Key type .llcrypt unregistered [ 977.443981] Key type ._llcrypt unregistered [ 984.779799] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 984.944768] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 984.952772] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 984.960771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 987.694966] Key type ._llcrypt registered [ 987.697903] Key type .llcrypt registered [ 987.932852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 987.955470] alg: No test for adler32 (adler32-zlib) [ 988.913295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.965820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 988.987878] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 990.935077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 990.955087] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 993.429993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 993.465337] LNet: Removed LNI 10.1.2.3@tcp [ 993.492056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 993.544555] LNet: Removed LNI 10.1.2.31@tcp [ 993.567627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 993.591892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 994.573569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 994.605968] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 995.871953] LNet: 39626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.885185] LNetError: 39626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 995.898493] LNet: Removed LNI 10.1.2.31@tcp [ 995.941252] LNet: Removed LNI 10.1.2.3@tcp [ 996.842425] Key type .llcrypt unregistered [ 996.848476] Key type ._llcrypt unregistered [ 1010.916943] Key type ._llcrypt registered [ 1010.919256] Key type .llcrypt registered [ 1012.443128] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 02:26:09 (1786947969) [ 1013.106859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1013.116848] alg: No test for adler32 (adler32-zlib) [ 1014.098348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.169554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1014.200535] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1015.076293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.115826] LNet: 40585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.127516] LNetError: 40585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.156349] LNet: Removed LNI 192.168.204.42@tcp [ 1015.177428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.250459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.42@tcp [ 1015.291346] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1016.077692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.42@tcp [ 1016.116781] LNet: Removed LNI 192.168.204.42@tcp [ 1016.181178] LNet: 40720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.013703] Key type .llcrypt unregistered [ 1017.023735] Key type ._llcrypt unregistered [ 1029.704705] Key type ._llcrypt registered [ 1029.711903] Key type .llcrypt registered [ 1031.528942] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 02:26:28 (1786947988) [ 1033.667254] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg442-server [ 1035.784633] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 02:26:32 (1786947992) [ 1036.528302] Key type .llcrypt unregistered [ 1036.532899] Key type ._llcrypt unregistered [ 1043.920023] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1044.207308] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1044.232409] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1044.239507] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1047.153692] Key type ._llcrypt registered [ 1047.160639] Key type .llcrypt registered [ 1047.394362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1047.406664] alg: No test for adler32 (adler32-zlib) [ 1048.398846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1048.485375] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1049.325991] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1052.002629] LNet: 42563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.017279] LNetError: 42563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.028441] LNet: Removed LNI 192.168.204.42@tcp1 [ 1052.043249] LNet: Removed LNI 10.1.2.3@tcp2 [ 1052.813758] Key type .llcrypt unregistered [ 1052.820282] Key type ._llcrypt unregistered [ 1067.779406] Key type ._llcrypt registered [ 1067.792741] Key type .llcrypt registered [ 1069.374700] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 02:27:05 (1786948025) [ 1069.904614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.938197] alg: No test for adler32 (adler32-zlib) [ 1070.871490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1070.949400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1070.978362] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1071.844944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.871375] LNet: 43538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.878436] LNetError: 43538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.892574] LNet: Removed LNI 192.168.204.42@tcp [ 1071.937810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.047183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1072.075988] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1081.588948] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 02:27:17 (1786948037) [ 1081.762421] LNet: 44124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.780433] LNetError: 44124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.809141] LNet: Removed LNI 192.168.204.42@tcp [ 1082.773492] Key type .llcrypt unregistered [ 1082.779191] Key type ._llcrypt unregistered [ 1089.607184] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1089.926475] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1089.938577] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1089.946707] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1092.764751] Key type ._llcrypt registered [ 1092.767268] Key type .llcrypt registered [ 1093.016854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1093.043310] alg: No test for adler32 (adler32-zlib) [ 1094.062037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.175244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1094.240563] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1095.228298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1095.251839] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1097.862199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1097.894421] LNet: 44875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.904535] LNetError: 44875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1097.913185] LNet: Removed LNI 10.1.2.3@tcp [ 1097.946248] LNet: Removed LNI 192.168.204.42@tcp [ 1097.961894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.021344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1098.039420] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1098.046703] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1098.575027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1098.597560] LNet: 44902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.603890] LNetError: 44902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1098.609701] LNetError: 44902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1098.615597] LNet: Removed LNI 10.1.2.3@tcp [ 1098.625988] LNet: Removed LNI 192.168.204.42@tcp [ 1098.665622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.739820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1098.763642] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1098.768268] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1099.388908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1099.426640] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 1100.745263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1100.772719] LNet: 45065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.783697] LNetError: 45065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1100.789794] LNetError: 45065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1100.795661] LNet: Removed LNI 10.1.2.3@tcp [ 1100.799739] LNet: Removed LNI 192.168.204.42@tcp [ 1100.827904] LNet: Removed LNI 192.168.204.42@tcp2 [ 1100.851364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.926345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1100.955915] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1100.959834] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1100.969749] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 1109.381545] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 02:27:46 (1786948066) [ 1109.493508] LNet: 45653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.507560] LNetError: 45653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1109.512889] LNetError: 45653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1109.518182] LNet: Removed LNI 10.1.2.3@tcp [ 1109.519647] LNet: Removed LNI 192.168.204.42@tcp [ 1109.536815] LNet: Removed LNI 192.168.204.42@tcp2 [ 1110.378539] Key type .llcrypt unregistered [ 1110.382696] Key type ._llcrypt unregistered [ 1117.447503] Key type ._llcrypt registered [ 1117.453776] Key type .llcrypt registered [ 1117.776984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1117.801436] alg: No test for adler32 (adler32-zlib) [ 1118.688968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1118.730549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1118.748629] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1119.827752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1119.851102] LNet: 46061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.856778] LNetError: 46061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.870916] LNet: Removed LNI 192.168.204.42@tcp [ 1119.887329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1119.955205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1119.984823] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1120.753751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1120.779924] LNet: 46092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.784755] LNetError: 46092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1120.817747] LNet: Removed LNI 192.168.204.42@tcp [ 1120.858205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1120.944262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1120.978137] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1121.674854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1121.744379] LNet: Removed LNI 192.168.204.42@tcp [ 1121.791266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1121.880767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1121.917341] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1122.713244] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1122.834408] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1122.845200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1122.859339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1125.405157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1125.449515] LNet: 46201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.464171] LNet: 46201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1125.491761] LNetError: 46201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.499435] LNetError: 46201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1125.517196] LNet: Removed LNI 192.168.204.42@tcp [ 1125.559709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.666388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1125.703043] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1134.781484] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 02:28:11 (1786948091) [ 1134.931730] LNet: 46792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.938837] LNetError: 46792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.969626] LNet: Removed LNI 192.168.204.42@tcp [ 1135.906670] Key type .llcrypt unregistered [ 1135.913933] Key type ._llcrypt unregistered [ 1143.622759] Key type ._llcrypt registered [ 1143.624919] Key type .llcrypt registered [ 1143.889525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1143.906647] alg: No test for adler32 (adler32-zlib) [ 1144.891548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1144.969342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1145.019755] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1145.951253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1145.970268] LNet: 47202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.975667] LNetError: 47202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.988810] LNet: Removed LNI 192.168.204.42@tcp [ 1146.028363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1146.125711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1146.162794] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1153.828336] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 02:28:30 (1786948110) [ 1154.152895] LNet: 47792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.165176] LNetError: 47792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.207657] LNet: Removed LNI 192.168.204.42@tcp [ 1155.250517] Key type .llcrypt unregistered [ 1155.263236] Key type ._llcrypt unregistered [ 1163.498934] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1163.775403] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1163.795086] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1163.810300] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1166.377319] Key type ._llcrypt registered [ 1166.380957] Key type .llcrypt registered [ 1166.573635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1166.590797] alg: No test for adler32 (adler32-zlib) [ 1167.535386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1167.614651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1167.651906] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1168.549754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1168.602847] LNet: 48250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.616603] LNetError: 48250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.647700] LNet: Removed LNI 192.168.204.42@tcp [ 1168.683569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.752206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1168.773830] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1175.760281] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 02:28:52 (1786948132) [ 1175.860343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1175.907913] LNet: 48842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.921417] LNetError: 48842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.944964] LNet: Removed LNI 192.168.204.42@tcp [ 1175.988404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1176.139877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1182.424906] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 02:28:59 (1786948139) [ 1182.750180] LNet: 49431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.459787] Key type .llcrypt unregistered [ 1183.463065] Key type ._llcrypt unregistered [ 1191.259377] Key type ._llcrypt registered [ 1191.263555] Key type .llcrypt registered [ 1191.503026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.522790] alg: No test for adler32 (adler32-zlib) [ 1192.487907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1192.586621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1192.606650] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1193.523380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1193.551816] LNet: 49843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.560899] LNetError: 49843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1193.578819] LNet: Removed LNI 192.168.204.42@tcp [ 1193.596726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1193.660663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1193.698354] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1201.050192] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 02:29:18 (1786948158) [ 1201.409742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1201.455571] LNet: 50437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.466209] LNetError: 50437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.497950] LNet: Removed LNI 192.168.204.42@tcp [ 1201.542960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1201.619494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1208.020683] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 02:29:25 (1786948165) [ 1209.830456] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1211.600978] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 02:29:28 (1786948168) [ 1213.191379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1214.877325] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 02:29:32 (1786948172) [ 1216.572081] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1218.454683] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 02:29:35 (1786948175) [ 1219.826873] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1221.482515] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 02:29:38 (1786948178) [ 1222.889903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1224.845224] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 02:29:41 (1786948181) [ 1225.099718] LNet: 52487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.985508] Key type .llcrypt unregistered [ 1225.992206] Key type ._llcrypt unregistered [ 1233.097753] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1233.269498] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1233.272082] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1233.279348] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1235.812096] Key type ._llcrypt registered [ 1235.814260] Key type .llcrypt registered [ 1235.981536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1235.996927] alg: No test for adler32 (adler32-zlib) [ 1236.912173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.000878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1237.036693] LNet: Added LNI 192.168.204.42@tcp [8/512/0/180] [ 1237.041715] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1237.044762] LNet: 52815:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1237.175729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1237.197700] LNet: 52834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.205965] LNetError: 52834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1237.212518] LNet: Removed LNI 10.1.2.3@tcp [ 1237.214656] LNetError: 52834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.232421] LNet: Removed LNI 192.168.204.42@tcp [ 1237.262712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.370051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1237.389780] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1237.460584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1237.487193] LNet: 52858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.495097] LNetError: 52858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.511641] LNet: Removed LNI 192.168.204.42@tcp [ 1237.529432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.626992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1237.661968] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1237.669898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1237.673116] LNet: 52866:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1237.774473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1237.813179] LNet: 52884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.825902] LNetError: 52884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1237.832682] LNet: Removed LNI 10.1.2.3@tcp [ 1237.836116] LNetError: 52884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.859834] LNet: Removed LNI 192.168.204.42@tcp [ 1237.879344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.949751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1237.973838] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1237.979029] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1237.982407] LNet: 52892:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1238.221823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1238.239744] LNet: 52919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.250551] LNetError: 52919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1238.257115] LNet: Removed LNI 10.1.2.3@tcp [ 1238.260715] LNetError: 52919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.274661] LNet: Removed LNI 192.168.204.42@tcp [ 1238.292195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.370742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1238.399440] LNet: Added LNI 192.168.204.42@tcp [8/256/0/182] [ 1238.406038] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1238.537647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1238.565995] LNet: 52947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.578593] LNetError: 52947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1238.590674] LNet: Removed LNI 10.1.2.3@tcp [ 1238.593200] LNetError: 52947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.614697] LNet: Removed LNI 192.168.204.42@tcp [ 1238.649564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.723745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1238.773933] LNet: Added LNI 192.168.204.42@tcp [8/256/0/182] [ 1238.779609] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1238.781108] LNet: 52955:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1238.879657] LNet: Removed LNI 10.1.2.3@tcp [ 1238.900409] LNet: Removed LNI 192.168.204.42@tcp [ 1239.475403] Key type .llcrypt unregistered [ 1239.478089] Key type ._llcrypt unregistered [ 1251.806194] Key type ._llcrypt registered [ 1251.809040] Key type .llcrypt registered [ 1253.354491] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 02:30:10 (1786948210) [ 1255.126525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1257.485523] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 02:30:14 (1786948214) [ 1258.156262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1258.179974] alg: No test for adler32 (adler32-zlib) [ 1259.191446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1259.284049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1259.324131] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1260.083896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.42 [ 1260.129186] LNet: 54224:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1260.146028] LNet: Removed LNI 192.168.204.42@tcp [ 1267.944318] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 02:30:25 (1786948225) [ 1268.040396] LNet: 54785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.053185] LNetError: 54785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.097653] LNet: Removed LNI 192.168.204.42@tcp [ 1268.946319] Key type .llcrypt unregistered [ 1268.948255] Key type ._llcrypt unregistered [ 1278.438237] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1278.599616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1280.214984] Key type ._llcrypt registered [ 1280.221823] Key type .llcrypt registered [ 1280.526527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1280.563230] alg: No test for adler32 (adler32-zlib) [ 1281.922514] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1290.039383] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 02:30:46 (1786948246) [ 1290.370715] LNet: 55706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.382121] LNetError: 55706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1290.403271] LNet: Removed LNI 10.1.2.3@tcp [ 1291.546680] Key type .llcrypt unregistered [ 1291.550804] Key type ._llcrypt unregistered [ 1298.344220] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1298.483672] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1298.488071] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1298.492621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1300.131513] Key type ._llcrypt registered [ 1300.133920] Key type .llcrypt registered [ 1300.263341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1300.274621] alg: No test for adler32 (adler32-zlib) [ 1301.331930] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1309.520655] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 02:31:06 (1786948266) [ 1311.079660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1313.549273] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 02:31:10 (1786948270) [ 1313.704714] LNet: 56879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.712444] LNetError: 56879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1313.766388] LNet: Removed LNI 10.1.2.3@tcp [ 1314.841918] Key type .llcrypt unregistered [ 1314.845095] Key type ._llcrypt unregistered [ 1321.522940] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1321.813798] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1321.816458] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1321.823055] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1324.621546] Key type ._llcrypt registered [ 1324.624835] Key type .llcrypt registered [ 1324.891248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1324.916459] alg: No test for adler32 (adler32-zlib) [ 1325.867541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.948628] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1335.409079] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 02:31:32 (1786948292) [ 1336.018908] LNet: 58122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.028762] LNetError: 58122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1336.055225] LNet: Removed LNI 10.1.2.3@tcp [ 1336.812943] Key type .llcrypt unregistered [ 1336.816337] Key type ._llcrypt unregistered [ 1343.408591] Key type ._llcrypt registered [ 1343.412366] Key type .llcrypt registered [ 1343.645576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1343.680981] alg: No test for adler32 (adler32-zlib) [ 1344.671694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1344.723396] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1353.683930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1354.591031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1354.634930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1354.677400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1354.702400] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1354.944929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1354.972732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1354.979530] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1355.003218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1355.037651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1355.064561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1355.095371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1355.123713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1355.155326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1355.171870] LNet: 58565:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1355.207970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1355.258809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1355.310541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1355.361723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1355.460519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1355.526501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1355.573515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1356.329679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1356.397886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1356.461749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1356.494317] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1356.785370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1356.822042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1356.846113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1356.871625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1356.908503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1356.980838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1357.032246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1357.068783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1357.091771] LNet: 58675:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1357.117613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1357.148772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1357.214742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1357.266165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1357.312626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1357.390516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1357.438887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1358.108122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1358.145315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1358.189619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1358.224648] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1358.231380] LNet: Skipped 1 previous similar message [ 1358.458863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1358.489389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1358.513553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1358.538345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1358.580389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1358.605737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1358.629318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1358.666996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1358.692348] LNet: 58784:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1358.725634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1358.759469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1358.797651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1358.837417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1358.878632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.916264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1358.955789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1359.661313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1359.698637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1359.723302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1359.926345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1359.971072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1360.009111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1360.042271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1360.085341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1360.121885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1360.156630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1360.181714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1360.215502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1360.253285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1360.303884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1360.366413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1360.394396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.431763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1360.456660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1360.981352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1361.020371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1361.067775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1361.098269] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1361.106217] LNet: Skipped 1 previous similar message [ 1361.322703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1361.347978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1361.386761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1361.412689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1361.441300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1361.468542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1361.503619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1361.549383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1361.579653] LNet: 59002:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1361.587155] LNet: 59002:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1361.615877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1361.655383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1361.694815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1361.742393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1361.786389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1361.829392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1361.867524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1362.546096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1362.596988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1362.649290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1362.920503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1362.956396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1362.987564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1363.022770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1363.071989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1363.151558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1363.202358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1363.257599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1363.297719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1363.336605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1363.362116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1363.391302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1363.440722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1363.482922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1363.543211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1363.834895] LNet: 59157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.839332] LNetError: 59157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.860771] LNet: Removed LNI 192.168.204.42@tcp [ 1364.364184] Key type .llcrypt unregistered [ 1364.367680] Key type ._llcrypt unregistered [ 1384.749638] Key type ._llcrypt registered [ 1384.752897] Key type .llcrypt registered [ 1386.406693] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 02:32:23 (1786948343) [ 1387.516326] Key type .llcrypt unregistered [ 1387.522495] Key type ._llcrypt unregistered [ 1396.540670] Key type ._llcrypt registered [ 1396.550551] Key type .llcrypt registered [ 1396.887761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1396.904693] alg: No test for adler32 (adler32-zlib) [ 1397.864245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1397.886519] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1406.287971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1418.605368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.655356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1418.678240] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1420.241222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1420.308369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 1 [ 1420.359405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1420.417696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 1 [ 1420.460842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1420.516729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 1 [ 1420.550618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1420.612591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 1 [ 1420.643960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1420.666056] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1420.673465] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1420.950165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1420.984607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1421.031909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1421.070121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1421.121672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1421.189136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1421.260082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1421.320496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1421.342296] LNet: 60633:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1421.365937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1421.431884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1421.489607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1421.524536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1421.563467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1421.627326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1421.682901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1422.281417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1422.353422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 1 [ 1422.410273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1422.465901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 1 [ 1422.514247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1422.571704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 1 [ 1422.637980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1422.725661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 1 [ 1422.780366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1422.814301] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1422.819896] LNet: Skipped 2 previous similar messages [ 1423.156101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1423.189848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1423.228196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1423.271485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1423.323959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1423.366240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1423.402806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1423.440497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1423.469752] LNet: 60755:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1423.483974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1423.508161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1423.531849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1423.559661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1423.600572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1423.650926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1423.714566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1424.270873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1424.301617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 1 [ 1424.341163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1424.377152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 1 [ 1424.429396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1424.479159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 1 [ 1424.516737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1424.544995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 1 [ 1424.576138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1424.600230] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1424.607856] LNet: Skipped 2 previous similar messages [ 1424.932906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1424.975609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1425.035447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1425.082486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1425.133119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1425.166717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1425.231595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1425.278328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1425.295557] LNet: 60876:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1425.319632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1425.348892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1425.386564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1425.419349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1425.445124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1425.478531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1425.507718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1425.987780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1426.029258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 1 [ 1426.074226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1426.132582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 1 [ 1426.171862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1426.204173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 1 [ 1426.244590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1426.286042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 1 [ 1426.324647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1426.613923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1426.653479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1426.684925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1426.724238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1426.754292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1426.785687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1426.820485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1426.842971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1426.878963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1426.911831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1426.986223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1427.039930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1427.089559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1427.131931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1427.176896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1427.794155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1427.828983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 1 [ 1427.875733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1427.916346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 1 [ 1427.955292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1427.993959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 1 [ 1428.030472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1428.079299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 1 [ 1428.127630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1428.154136] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1428.163480] LNet: Skipped 5 previous similar messages [ 1428.487469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1428.518304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1428.564222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1428.602435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1428.661053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1428.705912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1428.738661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1428.778495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1428.796804] LNet: 61119:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1428.810874] LNet: 61119:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1428.831502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1428.873738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1428.906671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1428.967723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1429.006759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1429.040391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1429.089293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1429.582283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1429.619461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_error -r 1 [ 1429.700816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1429.778743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_error -r 1 [ 1429.836050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1429.882474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_error -r 1 [ 1429.948420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1430.017850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_error -r 1 [ 1430.065320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1430.357412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1430.386725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1430.439122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1430.479709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1430.539522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1430.585863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1430.629176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1430.665385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1430.720284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1430.763678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1430.798697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1430.854475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1430.902329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1430.937797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1430.977411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1436.774376] LNet: 61320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.780590] LNetError: 61320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.791170] LNet: Removed LNI 192.168.204.42@tcp [ 1437.793113] LNetError: 61320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.809878] LNet: Removed LNI 192.168.204.42@tcp1 [ 1438.295638] Key type .llcrypt unregistered [ 1438.297572] Key type ._llcrypt unregistered [ 1455.892356] Key type ._llcrypt registered [ 1455.896690] Key type .llcrypt registered [ 1457.265956] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:33:34 (1786948414) [ 1458.554626] Key type .llcrypt unregistered [ 1458.557455] Key type ._llcrypt unregistered [ 1466.771009] Key type ._llcrypt registered [ 1466.775723] Key type .llcrypt registered [ 1467.109775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1467.161881] alg: No test for adler32 (adler32-zlib) [ 1468.115980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1468.147795] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1476.823748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1478.359477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1478.410656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1478.498697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1478.542669] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1478.995809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1479.040630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1479.107387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1479.204213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1479.274292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1479.434517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1479.525787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1479.609288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1479.643702] LNet: 62576:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1479.672736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1479.747268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1479.815167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1479.915825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1480.001189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1480.055278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1480.130972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1480.847136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1480.886406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1480.933456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1480.970897] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1481.347775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1481.390302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1481.414246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1481.444146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1481.486334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1481.529185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1481.611121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1481.687731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1481.721451] LNet: 62685:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1481.756434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1481.815666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1481.877754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1481.925777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1481.987416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1482.045824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1482.095516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1482.832462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1482.872576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1482.904935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1482.933905] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1483.315352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1483.373496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1483.412821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1483.474214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1483.518497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1483.568841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1483.607794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1483.644664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1483.676403] LNet: 62794:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1483.723671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1483.768388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1483.808929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1483.844381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1483.889274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1483.930823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1483.956819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1484.311543] LNet: 62840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.320888] LNetError: 62840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.334733] LNet: Removed LNI 192.168.204.42@tcp [ 1484.701494] Key type .llcrypt unregistered [ 1484.707019] Key type ._llcrypt unregistered [ 1504.258597] Key type ._llcrypt registered [ 1504.260172] Key type .llcrypt registered [ 1505.510333] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:34:22 (1786948462) [ 1506.189630] Key type .llcrypt unregistered [ 1506.194233] Key type ._llcrypt unregistered [ 1513.378084] Key type ._llcrypt registered [ 1513.380154] Key type .llcrypt registered [ 1513.749427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1513.770392] alg: No test for adler32 (adler32-zlib) [ 1514.723865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1514.760589] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1524.331819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1534.635671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.688802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1534.720592] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1536.164736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1536.202277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 1 [ 1536.238746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1536.277212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 1 [ 1536.307053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1536.338695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 1 [ 1536.380893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1536.438457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 1 [ 1536.485864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1536.512974] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1536.519929] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 1536.830206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1536.890877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1536.942945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1536.979270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1537.019204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1537.076319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1537.140841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1537.183985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1537.209678] LNet: 64312:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1537.241985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1537.270371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1537.297883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1537.331416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1537.357316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1537.396841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1537.434582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1537.485482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1538.019581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1538.102938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_error -r 1 [ 1538.181207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1538.270790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_error -r 1 [ 1538.360849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1538.436439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_error -r 1 [ 1538.502913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1538.588782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_error -r 1 [ 1538.702565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1538.745711] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1538.752429] LNet: Skipped 2 previous similar messages [ 1539.073510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1539.110990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1539.161224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1539.206505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1539.257435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1539.305290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1539.360508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1539.408821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1539.427755] LNet: 64435:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1539.462786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1539.497916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1539.562539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1539.613938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1539.660530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1539.699786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1539.736578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1539.786948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1540.507216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1540.587666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_timeout -r 1 [ 1540.670889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1540.721388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_timeout -r 1 [ 1540.764801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1540.798713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_timeout -r 1 [ 1540.840668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1540.880582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_timeout -r 1 [ 1540.956342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1540.985768] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1540.998518] LNet: Skipped 1 previous similar message [ 1541.269259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1541.310889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1541.352586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1541.388595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1541.436776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1541.474088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1541.521069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1541.563505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1541.589533] LNet: 64559:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1541.614954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1541.658574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1541.690255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1541.720718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1541.755693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1541.792321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1541.821614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1541.847547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1546.928850] LNet: 64641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.936836] LNetError: 64641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.943778] LNet: Removed LNI 192.168.204.42@tcp [ 1546.957065] LNet: Removed LNI 192.168.204.42@tcp1 [ 1547.402510] Key type .llcrypt unregistered [ 1547.410811] Key type ._llcrypt unregistered [ 1564.657537] Key type ._llcrypt registered [ 1564.659738] Key type .llcrypt registered [ 1566.173812] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:35:22 (1786948522) [ 1567.894525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1569.613182] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:35:26 (1786948526) [ 1571.075445] Key type .llcrypt unregistered [ 1571.082098] Key type ._llcrypt unregistered [ 1578.080411] Key type ._llcrypt registered [ 1578.091212] Key type .llcrypt registered [ 1578.361357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1578.381517] alg: No test for adler32 (adler32-zlib) [ 1579.391291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1579.414666] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1587.429988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1588.274404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1588.358053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1588.423895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1588.444275] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1588.640389] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1588.746793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1588.777814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1588.810548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1588.841315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1588.868368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1588.894933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1588.919778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1588.948464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1588.968429] LNet: 66191:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1588.990976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1589.041721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1589.110380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1589.171666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1589.241415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1589.304283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1589.656651] LNet: 66234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.681281] LNetError: 66234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.699792] LNet: Removed LNI 192.168.204.42@tcp [ 1590.170793] Key type .llcrypt unregistered [ 1590.174164] Key type ._llcrypt unregistered [ 1610.605109] Key type ._llcrypt registered [ 1610.614873] Key type .llcrypt registered [ 1610.911516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1610.933279] alg: No test for adler32 (adler32-zlib) [ 1611.998399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1612.044185] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1621.093963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1631.016891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.072792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1631.102087] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1632.571341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1632.613945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e network_timeout -r 1 [ 1632.662676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1632.696322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e network_timeout -r 1 [ 1632.739803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1632.769486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e network_timeout -r 1 [ 1632.798662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1632.854026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e network_timeout -r 1 [ 1632.913190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1632.940352] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1633.151237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1633.184535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1633.212164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1633.238605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1633.262562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1633.281362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1633.309964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1633.357755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1633.381926] LNet: 67120:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1633.409718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1633.460229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1633.492945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1633.524493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1633.550513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1633.570342] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1633.570688] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1633.575076] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 1633.595501] LNet: Skipped 1 previous similar message [ 1633.607682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1635.616781] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1637.672553] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1639.194925] LNet: 67198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.200511] LNetError: 67198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.210852] LNet: Removed LNI 192.168.204.42@tcp [ 1639.229593] LNet: Removed LNI 192.168.204.42@tcp1 [ 1639.735896] Key type .llcrypt unregistered [ 1639.737600] Key type ._llcrypt unregistered [ 1657.676646] Key type ._llcrypt registered [ 1657.678597] Key type .llcrypt registered [ 1659.089286] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:36:55 (1786948615) [ 1659.419487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1659.433764] alg: No test for adler32 (adler32-zlib) [ 1660.421511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.492556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.538568] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1661.251933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1661.287133] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1662.283534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1662.352739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1662.421274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1662.472275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1662.539386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1662.590526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1662.620709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 1662.640814] LNetError: 68299:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1663.520535] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1663.523439] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1665.567732] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1666.203355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1669.663367] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1669.878036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1669.911341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1669.973955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.997749] LNet: 68406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.003272] LNetError: 68406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.011787] LNet: Removed LNI 192.168.204.42@tcp [ 1670.033071] LNet: Removed LNI 192.168.204.42@tcp1 [ 1670.073791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.143550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1670.184133] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1671.009976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1671.036146] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1672.008588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1672.041448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1672.116777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1672.181145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1672.224743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1672.266325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1672.312266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1672.352352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 1672.377399] LNetError: 68679:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1673.183801] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1675.999537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1679.329946] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1679.338028] LNet: Skipped 1 previous similar message [ 1684.119974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1684.151550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1684.180921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1691.424556] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:37:28 (1786948648) [ 1691.772304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.816879] LNet: 69396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.848629] LNetError: 69396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.876888] LNetError: 69396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1691.903384] LNet: Removed LNI 192.168.204.42@tcp [ 1691.964628] LNet: Removed LNI 192.168.204.42@tcp1 [ 1692.004219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.067164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1692.100150] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1693.188453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1693.209808] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1694.786812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1694.954464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1695.037449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1695.102365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1695.172645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1695.235343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1695.292313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 0 [ 1695.392852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1696.160383] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.42@tcp [ 1696.161532] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1696.174647] LNet: Skipped 1 previous similar message [ 1711.109415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1711.306933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1711.337516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1711.372983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 500 [ 1711.623890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1718.510903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1718.620750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.658701] LNet: 69890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.678296] LNetError: 69890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.692879] LNetError: 69890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1718.706128] LNet: Removed LNI 192.168.204.42@tcp [ 1718.751635] LNet: Removed LNI 192.168.204.42@tcp1 [ 1718.783420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.896715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1718.964227] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1719.826251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1719.843075] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1721.049427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1721.118441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1721.207788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1721.256368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1721.319200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1721.397446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1721.449720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1721.500139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 0 [ 1722.985790] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1723.010020] LNet: Skipped 3 previous similar messages [ 1724.038791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1733.309644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1733.359106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1733.404238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1742.417538] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:38:19 (1786948699) [ 1742.954072] LNet: 70937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.959744] LNetError: 70937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.964892] LNetError: 70937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1742.970297] LNet: Removed LNI 192.168.204.42@tcp [ 1742.979075] LNet: Removed LNI 192.168.204.42@tcp1 [ 1743.518825] Key type .llcrypt unregistered [ 1743.521848] Key type ._llcrypt unregistered [ 1750.534521] Key type ._llcrypt registered [ 1750.543818] Key type .llcrypt registered [ 1750.876607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1750.908254] alg: No test for adler32 (adler32-zlib) [ 1751.878618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1751.923841] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1762.988251] Lustre: DEBUG MARKER: Initial discovery [ 1763.114061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 1766.116350] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1766.211772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1766.299363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 1766.326831] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1766.342953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1766.399532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.142@tcp -r 1 -m GET -l 3 [ 1766.450278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.142@tcp -r 1 -m GET -e local_error [ 1766.481859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.142@tcp -r 1 -m PUT -l 6 [ 1767.327400] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1768.021385] Lustre: DEBUG MARKER: Force oleg442-client.virtnet to discover 192.168.204.142@tcp (in background) [ 1768.488171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 1770.114809] Lustre: DEBUG MARKER: Force oleg442-server to discover 192.168.204.42@tcp [ 1771.555922] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1773.282845] Lustre: DEBUG MARKER: Wait for 71706 [ 1774.715526] Lustre: DEBUG MARKER: Finished wait on 71706 [ 1774.945557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1774.977529] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1774.991621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1775.253990] LNet: 72158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.261790] LNetError: 72158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.288423] LNet: Removed LNI 192.168.204.42@tcp [ 1775.814094] Key type .llcrypt unregistered [ 1775.816895] Key type ._llcrypt unregistered [ 1796.207400] Key type ._llcrypt registered [ 1796.209206] Key type .llcrypt registered [ 1797.831591] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:39:14 (1786948754) [ 1798.174483] Key type .llcrypt unregistered [ 1798.176317] Key type ._llcrypt unregistered [ 1805.479079] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1805.706340] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1805.714757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1805.720743] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1808.375714] Key type ._llcrypt registered [ 1808.381480] Key type .llcrypt registered [ 1808.698153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1808.720790] alg: No test for adler32 (adler32-zlib) [ 1809.730419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.813430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1809.874991] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1810.836920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1810.858618] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1821.114702] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:39:37 (1786948777) [ 1821.324462] LNet: 74236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.339564] LNetError: 74236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1821.349429] LNet: Removed LNI 10.1.2.3@tcp [ 1821.381859] LNet: Removed LNI 192.168.204.42@tcp [ 1822.238637] Key type .llcrypt unregistered [ 1822.240991] Key type ._llcrypt unregistered [ 1829.477634] Key type ._llcrypt registered [ 1829.484303] Key type .llcrypt registered [ 1829.797139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1829.825165] alg: No test for adler32 (adler32-zlib) [ 1830.720248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.787961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1830.842412] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1831.678349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1831.719869] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1832.970265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.42@tcp [ 1833.015540] LNetError: Unexpected error -22 connecting to 192.168.204.42@tcp at host 192.168.204.42:988 [ 1840.947860] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:39:57 (1786948797) [ 1841.246334] LNet: 75377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.251989] LNetError: 75377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1841.260910] LNet: Removed LNI 10.1.2.3@tcp [ 1841.270606] LNet: Removed LNI 192.168.204.42@tcp [ 1842.062435] Key type .llcrypt unregistered [ 1842.064151] Key type ._llcrypt unregistered [ 1849.821687] Key type ._llcrypt registered [ 1849.832952] Key type .llcrypt registered [ 1850.092900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1850.111733] alg: No test for adler32 (adler32-zlib) [ 1851.163687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.241301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1851.277705] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1852.272022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1852.304180] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 1853.344955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.42@tcp1 --nid 192.168.204.42@tcp2 [ 1858.506260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp1 192.168.204.42@tcp2 [ 1858.536902] LNetError: 76216:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.42@tcp1 to 192.168.204.42@tcp2 - Route table may be misconfigured [ 1858.583711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp2 192.168.204.42@tcp1 [ 1865.116176] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:40:22 (1786948822) [ 1865.510552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1865.551436] LNet: 76789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.562813] LNetError: 76789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.578967] LNet: Removed LNI 192.168.204.42@tcp1 [ 1866.605082] LNet: Removed LNI 192.168.204.42@tcp2 [ 1866.626241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.689769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1866.731796] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1867.508758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1867.527516] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1868.450062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1868.504445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.42@tcp [ 1868.554797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp1 192.168.204.42@tcp1 [ 1868.625688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp -d 192.168.204.42@tcp -e network_timeout [ 1868.693037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -e network_timeout [ 1868.764344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -e network_timeout [ 1868.815770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -e network_timeout [ 1868.859278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1868.915383] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1868.938866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1869.076630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1869.208545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1869.727733] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1876.192121] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:40:33 (1786948833) [ 1876.665700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.697746] LNet: 77651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.708476] LNetError: 77651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1876.713439] LNetError: 77651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1876.717963] LNet: Removed LNI 192.168.204.42@tcp [ 1877.735880] LNet: Removed LNI 192.168.204.42@tcp1 [ 1877.813571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.012460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1878.456877] LNet: 77692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.039541] Key type .llcrypt unregistered [ 1879.044792] Key type ._llcrypt unregistered [ 1891.687279] Key type ._llcrypt registered [ 1891.691817] Key type .llcrypt registered [ 1893.261080] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:40:49 (1786948849) [ 1893.808058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1893.840707] alg: No test for adler32 (adler32-zlib) [ 1894.836478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.454444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1898.475471] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:40:55 (1786948855) [ 1898.775178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.791879] LNet: 78791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.838419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.937299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1898.973778] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1899.962768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1900.008102] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1901.110470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1901.180317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp1 [ 1901.239771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp1 [ 1907.718994] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:41:04 (1786948864) [ 1909.591939] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg442-server [ 1911.522774] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:41:08 (1786948868) [ 1913.673279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg442-server [ 1915.399174] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:41:12 (1786948872) [ 1917.789136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg442-server [ 1919.630470] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:41:16 (1786948876) [ 1921.906798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg442-server [ 1924.022337] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:41:20 (1786948880) [ 1926.178319] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg442-server [ 1928.080302] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:41:25 (1786948885) [ 1930.117849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg442-server [ 1932.081087] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:41:29 (1786948889) [ 1933.895915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg442-server [ 1935.673065] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:41:32 (1786948892) [ 1937.378548] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg442-server [ 1939.531227] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:41:36 (1786948896) [ 1941.586567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg442-server [ 1943.676372] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:41:40 (1786948900) [ 1943.833194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.868388] LNet: 82668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.882884] LNetError: 82668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.927264] LNet: Removed LNI 192.168.204.42@tcp [ 1944.942639] LNetError: 82668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.979872] LNet: Removed LNI 192.168.204.42@tcp1 [ 1945.034533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.121747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1945.149923] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1946.088874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1946.254779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1946.504882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.537360] LNet: 82816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.546347] LNetError: 82816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.566591] LNet: Removed LNI 192.168.204.42@tcp [ 1946.603888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.664130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1946.681768] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1947.418972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1947.608661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1948.712253] LNet: Removed LNI 192.168.204.42@tcp [ 1948.779199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.855540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.880678] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1949.926874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1950.032338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1950.304589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1950.319713] LNet: 83109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.327454] LNet: 83109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1950.334456] LNetError: 83109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1950.343474] LNetError: 83109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1950.357715] LNet: Removed LNI 192.168.204.42@tcp [ 1950.385185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.468453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1950.494233] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1951.322397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1951.565687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1952.619520] LNet: Removed LNI 192.168.204.42@tcp [ 1952.685694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.768296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1952.784466] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1953.319259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1953.375175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1953.695939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1954.791460] LNet: Removed LNI 192.168.204.42@tcp [ 1954.801442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.838551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1954.853962] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1955.446254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1955.622847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.644388] LNet: 83546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.648656] LNet: 83546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1955.654890] LNetError: 83546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1955.666709] LNetError: 83546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1955.682793] LNet: Removed LNI 192.168.204.42@tcp [ 1955.703605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.737899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1955.760703] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1956.600995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1956.729333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1957.053410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1958.119706] LNet: Removed LNI 192.168.204.42@tcp [ 1958.186313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.271334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1958.300020] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1959.397685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1959.627122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1960.691771] LNet: Removed LNI 192.168.204.42@tcp [ 1960.714515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.770875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1960.802035] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1961.810143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1961.933907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1962.277746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.373527] LNet: Removed LNI 192.168.204.42@tcp [ 1963.414782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.548517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1963.577820] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1964.379283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1964.770614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.795850] LNet: 84132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.807465] LNet: 84132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1964.819745] LNetError: 84132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.830538] LNetError: 84132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1964.850293] LNet: Removed LNI 192.168.204.42@tcp [ 1964.872282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.921176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1964.938591] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1965.640925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1965.789721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1967.254058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.295242] LNet: Removed LNI 192.168.204.42@tcp [ 1968.320641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1968.414122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1968.459331] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1969.468725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1969.719403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1970.792560] LNet: Removed LNI 192.168.204.42@tcp [ 1970.810244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.873604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1970.894706] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1971.573972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1971.613923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1971.654827] LNet: Removed LNI 192.168.204.42@tcp [ 1971.667049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1971.729303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1971.820895] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1972.542993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1972.714262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.741945] LNet: Removed LNI 192.168.204.42@tcp [ 1972.770314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.886724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1973.063437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.131817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.223678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1973.383959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.450811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.549685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1973.652301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.681894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.727645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1973.817673] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1974.713698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.741784] LNet: Removed LNI 192.168.204.42@tcp [ 1974.772564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.860702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1984.487892] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:42:19 (1786948939) [ 1984.894260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1984.941725] LNet: 85455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.952392] LNet: 85455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1984.996387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.099790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1985.140920] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1987.098196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1987.147245] LNetError: 85593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1987.156529] LNetError: 85593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1987.185916] LNet: Removed LNI 192.168.204.42@tcp231 [ 1987.220594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.386858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1987.435415] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1987.445089] LNet: 85601:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1988.247845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1988.284454] LNet: Removed LNI 192.168.204.42@tcp231 [ 1988.322773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1988.360743] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1989.800767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.833857] LNet: Removed LNI 192.168.204.42@tcp231 [ 1989.879250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.956614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1989.988148] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/0] [ 1989.997335] LNet: 85754:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1990.617585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1990.675842] LNet: Removed LNI 192.168.204.42@tcp231 [ 1990.703886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1990.743892] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/0] [ 1992.674123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.741652] LNet: Removed LNI 192.168.204.42@tcp231 [ 1992.785711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.934958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1992.971906] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/60] [ 1992.981771] LNet: 85905:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1993.574124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1993.615913] LNet: Removed LNI 192.168.204.42@tcp231 [ 1993.636421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1993.666539] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/60] [ 1995.129329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.162597] LNet: Removed LNI 192.168.204.42@tcp231 [ 1995.205947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1995.340207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1995.412552] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1995.417541] LNet: 86057:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1995.697161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1995.772230] LNet: Removed LNI 192.168.204.42@tcp231 [ 1995.804961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1995.844146] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 2006.678149] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:42:42 (1786948962) [ 2006.847160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2006.924155] LNet: Removed LNI 192.168.204.42@tcp231 [ 2006.955337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.043429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2007.111494] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 2008.176415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2008.237922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2008.290356] LNet: Removed LNI 192.168.204.42@tcp232 [ 2008.314843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2008.384352] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 2009.702773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2009.832381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2009.887041] LNet: Removed LNI 192.168.204.42@tcp232 [ 2009.922793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2009.976578] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 2010.776056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2010.823603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2010.849431] LNet: Removed LNI 192.168.204.42@tcp232 [ 2010.872267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2010.903201] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 2011.807986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2011.854329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2011.895325] LNet: Removed LNI 192.168.204.42@tcp232 [ 2011.909758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2019.921192] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:42:56 (1786948976) [ 2020.612791] LNet: 87898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.623535] LNet: 87898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2021.651864] Key type .llcrypt unregistered [ 2021.653423] Key type ._llcrypt unregistered [ 2031.865064] Key type ._llcrypt registered [ 2031.869040] Key type .llcrypt registered [ 2032.020405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2032.043175] alg: No test for adler32 (adler32-zlib) [ 2033.023341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2033.058579] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2042.680563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2053.085420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.170287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2053.201540] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2054.893255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 2 [ 2054.941730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 2 [ 2054.971898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 2 [ 2055.015448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 2 [ 2055.085639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_interrupt -r 2 [ 2055.124884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 2 [ 2055.163660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_interrupt -r 2 [ 2055.226188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 2 [ 2055.267950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2055.322052] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2055.333749] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 2055.745841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2055.816363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2055.873720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2055.959414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2056.052895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2056.113772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2056.141916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2056.193054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2056.248338] LNet: 88567:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2056.269523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2056.299743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2056.351560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2056.388522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2056.419302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2056.454451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2057.020440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 2 [ 2057.061331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 2 [ 2057.101198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 2 [ 2057.161583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 2 [ 2057.197404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_dropped -r 2 [ 2057.255796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 2 [ 2057.306443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_dropped -r 2 [ 2057.369486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 2 [ 2057.430723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2057.460328] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2057.707385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2057.732377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2057.756760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2057.787514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2057.813871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2057.836809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2057.890857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2057.942465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2057.959840] LNet: 88687:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2057.986923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2058.024235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2058.060270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2058.102581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2058.161596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2058.212646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2058.808236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 2 [ 2058.848135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 2 [ 2058.880847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 2 [ 2058.917636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 2 [ 2058.956950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_aborted -r 2 [ 2058.997240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 2 [ 2059.069523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_aborted -r 2 [ 2059.117444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 2 [ 2059.156971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2059.183571] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2059.460899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2059.497819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2059.536310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2059.580377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2059.610744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2059.638632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2059.676558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2059.732953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2059.749975] LNet: 88808:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2059.773252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2059.813425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2059.868316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2059.915930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2059.965601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2060.029291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2060.743938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 2 [ 2060.780693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 2 [ 2060.842922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 2 [ 2060.898181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 2 [ 2060.937848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_no_route -r 2 [ 2060.972937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 2 [ 2061.010100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_no_route -r 2 [ 2061.067392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 2 [ 2061.156800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2061.184210] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2061.544625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2061.602944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2061.672458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2061.753708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2061.803950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2061.852874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2061.892457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2061.930591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2061.949490] LNet: 88928:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2061.989289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2062.039487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2062.087518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2062.138565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2062.192971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2062.226262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2062.864425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 2 [ 2062.930332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 2 [ 2062.963695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 2 [ 2063.033542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 2 [ 2063.093085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_timeout -r 2 [ 2063.164394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 2 [ 2063.224635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_timeout -r 2 [ 2063.304658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 2 [ 2063.363954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2063.735904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2063.767922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2063.802848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2063.833651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2063.868328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2063.909575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2063.961270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2064.013484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2064.066657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2064.104484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2064.150671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2064.200619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2064.233070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2064.267380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2065.000148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 2 [ 2065.076726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 2 [ 2065.174296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 2 [ 2065.213684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 2 [ 2065.251282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_dropped -r 2 [ 2065.305545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 2 [ 2065.347471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_dropped -r 2 [ 2065.391785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 2 [ 2065.437570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2065.455756] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2065.463553] LNet: Skipped 1 previous similar message [ 2065.466397] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 2065.831892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2065.862908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2065.919748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2065.963765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2065.986690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2066.015764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2066.039619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2066.061474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2066.076534] LNet: 89168:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2066.080946] LNet: 89168:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2066.094836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2066.120826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2066.143993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2066.169791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2066.194515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2066.253534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2072.261225] LNet: 89247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.275723] LNetError: 89247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2072.284786] LNet: Removed LNI 192.168.204.42@tcp [ 2072.294680] LNet: Removed LNI 192.168.204.42@tcp1 [ 2072.771613] Key type .llcrypt unregistered [ 2072.773967] Key type ._llcrypt unregistered [ 2090.037154] Key type ._llcrypt registered [ 2090.039475] Key type .llcrypt registered [ 2091.427032] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:44:08 (1786949048) [ 2091.883897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2091.905808] alg: No test for adler32 (adler32-zlib) [ 2092.865651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2092.923387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2092.944807] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2093.783362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2093.813640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 2093.846390] LNetError: 90227:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 2093.919291] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 2093.940379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.42@tcp [ 2094.045313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2101.120108] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:44:18 (1786949058) [ 2101.341753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2101.373229] LNet: 90816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.391054] LNetError: 90816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.423202] LNet: Removed LNI 192.168.204.42@tcp [ 2101.461523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.541035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2101.576617] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2102.250874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2102.273041] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2103.265211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.42@tcp [ 2103.290260] LNetError: 91084:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 2103.589742] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 2104.557206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.42@tcp [ 2112.743380] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:44:29 (1786949069) [ 2115.421645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg442-server [ 2117.488115] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:44:34 (1786949074) [ 2120.158664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg442-server [ 2122.525890] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:44:39 (1786949079) [ 2124.806850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg442-server [ 2126.831866] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:44:43 (1786949083) [ 2127.154862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.188253] LNet: 92708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.198182] LNetError: 92708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.204326] LNet: Removed LNI 192.168.204.42@tcp [ 2127.213123] LNet: Removed LNI 192.168.204.42@tcp1 [ 2127.239032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.307228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.336273] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2128.153656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2128.225576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2128.294454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2128.419842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2128.464319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2128.519722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2136.513617] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:44:53 (1786949093) [ 2136.688906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.737534] LNet: 93429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.754079] LNetError: 93429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.769950] LNetError: 93429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2136.792702] LNet: Removed LNI 192.168.204.42@tcp [ 2136.833685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.925702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2136.969379] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2144.740798] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:45:01 (1786949101) [ 2144.937947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.970926] LNet: 94133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.986856] LNetError: 94133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2145.018244] LNet: Removed LNI 192.168.204.42@tcp [ 2145.043542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.123229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2145.153921] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2153.511794] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:45:10 (1786949110) [ 2155.451718] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2157.664255] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:45:14 (1786949114) [ 2158.378918] LNet: 95172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.397166] LNetError: 95172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.420098] LNet: Removed LNI 192.168.204.42@tcp [ 2159.238044] Key type .llcrypt unregistered [ 2159.243786] Key type ._llcrypt unregistered [ 2167.342285] Key type ._llcrypt registered [ 2167.345277] Key type .llcrypt registered [ 2167.699591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2167.719504] alg: No test for adler32 (adler32-zlib) [ 2168.930533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2169.023927] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2177.848423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2184.120893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.142@tcp [ 2184.662362] LNet: 95599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.670571] LNetError: 95599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2184.693519] LNet: Removed LNI 192.168.204.42@tcp [ 2185.248560] Key type .llcrypt unregistered [ 2185.254565] Key type ._llcrypt unregistered [ 2199.336404] Key type ._llcrypt registered [ 2199.344096] Key type .llcrypt registered [ 2201.006756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2203.055363] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2205.396197] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:46:02 (1786949162) [ 2206.875634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2209.098689] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:46:05 (1786949165) [ 2211.448424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg442-server [ 2213.629767] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:46:10 (1786949170) [ 2215.919111] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg442-server [ 2217.932977] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:46:14 (1786949174) [ 2220.331304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2220.358084] alg: No test for adler32 (adler32-zlib) [ 2221.647583] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 2221.916747] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2223.615258] Key type lgssc registered [ 2225.342279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2248.606175] Key type lgssc unregistered [ 2248.907604] LNet: 98779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.915581] LNetError: 98779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.942357] LNet: Removed LNI 192.168.204.42@tcp [ 2249.945417] Key type .llcrypt unregistered [ 2249.950147] Key type ._llcrypt unregistered [ 2267.957761] Key type ._llcrypt registered [ 2267.959948] Key type .llcrypt registered [ 2269.778266] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:47:06 (1786949226) [ 2270.490564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2270.504483] alg: No test for adler32 (adler32-zlib) [ 2271.583614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.676406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2271.755514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2271.816293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2271.873880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2279.474527] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:47:16 (1786949236) [ 2279.619518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2279.678365] LNet: 100190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.726846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.834368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.871228] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2281.119612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.143379] LNet: 100340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.155057] LNetError: 100340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.180886] LNet: Removed LNI 192.168.204.42@tcp [ 2281.199499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.284627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2281.331187] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2282.038637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.090465] LNetError: 100368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.115910] LNet: Removed LNI 192.168.204.42@tcp [ 2282.174096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.241358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2282.283735] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2290.065542] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:47:27 (1786949247) [ 2290.201037] LNet: 100954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.211383] LNet: 100954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2290.230341] LNetError: 100954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.273960] LNet: Removed LNI 192.168.204.42@tcp [ 2291.031629] Key type .llcrypt unregistered [ 2291.033513] Key type ._llcrypt unregistered [ 2298.353378] Key type ._llcrypt registered [ 2298.364971] Key type .llcrypt registered [ 2298.780329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2298.810497] alg: No test for adler32 (adler32-zlib) [ 2299.846912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.985180] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2299.988455] LNet: *** cfs_fail_loc=e003, val=100*** [ 2300.958238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2300.980082] LNet: *** cfs_fail_loc=e003, val=100*** [ 2300.986674] LNet: Skipped 2 previous similar messages [ 2301.007442] Ping reply received with mapping: [ 2301.010775] found metadata reply NI[192.168.204.42@tcp] [ 2301.017651] 2a cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2301.025834] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2301.030851] 5a 5a 5a 5a 5a 5a 5a 5a [ 2301.190330] LNet: 101361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.197169] LNetError: 101361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.252482] LNet: Removed LNI 192.168.204.42@tcp [ 2303.063457] Key type .llcrypt unregistered [ 2303.067514] Key type ._llcrypt unregistered [ 2316.589739] Key type ._llcrypt registered [ 2316.593210] Key type .llcrypt registered [ 2318.095581] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:47:54 (1786949274) [ 2318.921444] Key type .llcrypt unregistered [ 2318.929315] Key type ._llcrypt unregistered [ 2327.587876] Key type ._llcrypt registered [ 2327.593949] Key type .llcrypt registered [ 2327.982661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2328.005671] alg: No test for adler32 (adler32-zlib) [ 2329.401253] LNetError: 102419:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2330.020122] Key type .llcrypt unregistered [ 2330.023353] Key type ._llcrypt unregistered [ 2336.642390] Key type ._llcrypt registered [ 2336.645557] Key type .llcrypt registered [ 2338.297289] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:48:14 (1786949294) [ 2339.082320] Key type .llcrypt unregistered [ 2339.088927] Key type ._llcrypt unregistered [ 2347.042173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2347.048659] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2349.859615] Key type ._llcrypt registered [ 2349.863209] Key type .llcrypt registered [ 2350.113479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2350.128079] alg: No test for adler32 (adler32-zlib) [ 2351.073695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.182353] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2362.662233] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:48:39 (1786949319) [ 2362.860637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.900564] LNet: 104206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.912177] LNetError: 104206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2362.941615] LNet: Removed LNI 10.1.2.3@tcp [ 2363.003260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.097859] Key type .llcrypt unregistered [ 2364.100382] Key type ._llcrypt unregistered [ 2372.393198] Key type ._llcrypt registered [ 2372.404528] Key type .llcrypt registered [ 2372.672530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2372.689923] alg: No test for adler32 (adler32-zlib) [ 2373.679647] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2373.830592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2374.115351] LNet: 104503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.128780] LNetError: 104503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2374.154281] LNet: Removed LNI 192.168.204.42@tcp [ 2375.147703] Key type .llcrypt unregistered [ 2375.159032] Key type ._llcrypt unregistered [ 2389.071709] Key type ._llcrypt registered [ 2389.074058] Key type .llcrypt registered [ 2390.600949] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:49:07 (1786949347) [ 2391.799500] Key type .llcrypt unregistered [ 2391.804539] Key type ._llcrypt unregistered [ 2400.357670] Key type ._llcrypt registered [ 2400.364402] Key type .llcrypt registered [ 2400.711154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2400.750216] alg: No test for adler32 (adler32-zlib) [ 2410.036560] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:49:26 (1786949366) [ 2410.308046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.379123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.451212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2410.477234] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2411.390643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2411.446247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2411.480761] LNet: Removed LNI 192.168.204.42@tcp [ 2411.796753] LNet: 106262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2412.163445] Key type .llcrypt unregistered [ 2412.167746] Key type ._llcrypt unregistered [ 2425.384609] Key type ._llcrypt registered [ 2425.388720] Key type .llcrypt registered [ 2426.962933] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:49:43 (1786949383) [ 2427.525230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2427.554608] alg: No test for adler32 (adler32-zlib) [ 2428.585525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2428.661597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2428.690716] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2429.542463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2429.592214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.42@tcp [ 2429.627048] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2429.634378] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2429.643280] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2429, 0, 0 [ 2429.654538] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2429.662280] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2429.671817] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2429.678489] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2429, 0, 0 [ 2429.685582] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2429.690290] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2429.694531] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2429.699795] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2429, 0, 2479 [ 2429.705345] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2429.709403] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2429.712838] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2429.718252] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2429, 0, 2479 [ 2429.725778] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2436.824523] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:49:54 (1786949394) [ 2437.560774] LNet: 107805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.569559] LNetError: 107805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.629640] LNet: Removed LNI 192.168.204.42@tcp [ 2439.534962] Key type .llcrypt unregistered [ 2439.541382] Key type ._llcrypt unregistered [ 2447.476567] Key type ._llcrypt registered [ 2447.480778] Key type .llcrypt registered [ 2447.711728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2447.728216] alg: No test for adler32 (adler32-zlib) [ 2448.676610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2448.699488] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2457.007693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2467.908764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2467.966366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2467.998034] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2469.185638] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2469.299309] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2469.309397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2469.327668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2471.820771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2471.857794] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2484.595979] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:50:41 (1786949441) [ 2484.792642] LNet: 109312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.803727] LNetError: 109312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.814168] LNet: Removed LNI 192.168.204.42@tcp [ 2484.816328] LNet: Removed LNI 192.168.204.42@tcp1 [ 2484.848566] LNet: Removed LNI 10.1.2.3@tcp99 [ 2485.801764] Key type .llcrypt unregistered [ 2485.805436] Key type ._llcrypt unregistered [ 2493.473668] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2493.479446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2496.132215] Key type ._llcrypt registered [ 2496.137224] Key type .llcrypt registered [ 2496.426035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2496.442317] alg: No test for adler32 (adler32-zlib) [ 2497.339224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2497.394135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2497.413938] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2498.113807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2498.135411] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2500.228743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2500.268492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2500.319770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2500.347925] LNetError: 110068:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2500.411541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2500.467294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2500.989978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2501.115859] LNet: 110088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.122848] LNetError: 110088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2501.128302] LNet: Removed LNI 10.1.2.3@tcp [ 2502.186076] LNet: Removed LNI 192.168.204.42@tcp [ 2503.120454] Key type .llcrypt unregistered [ 2503.125437] Key type ._llcrypt unregistered [ 2515.816059] Key type ._llcrypt registered [ 2515.817465] Key type .llcrypt registered [ 2517.277803] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:51:14 (1786949474) [ 2517.746466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2517.819683] alg: No test for adler32 (adler32-zlib) [ 2518.782515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2518.837798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2518.873394] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2519.590860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2519.674561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg442-client@tcp [ 2526.986207] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:51:23 (1786949483) [ 2527.181085] LNet: 111625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2527.196858] LNetError: 111625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2527.234967] LNet: Removed LNI 192.168.204.42@tcp [ 2527.842637] Key type .llcrypt unregistered [ 2527.845972] Key type ._llcrypt unregistered [ 2534.368451] Key type ._llcrypt registered [ 2534.374726] Key type .llcrypt registered [ 2534.708107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2534.742043] alg: No test for adler32 (adler32-zlib) [ 2535.661352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2535.703960] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2535.742300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2542.609697] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:51:39 (1786949499) [ 2544.430436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2546.421178] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:51:43 (1786949503) [ 2548.340421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2549.891592] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:51:47 (1786949507) [ 2550.374540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2550.400634] LNet: 113050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2550.413582] LNetError: 113050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2550.427567] LNet: Removed LNI 192.168.204.42@tcp [ 2550.445690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2550.485887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2550.503790] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2551.392156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2551.464626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2558.349388] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:51:55 (1786949515) [ 2558.515966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2558.540130] LNet: 113758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2558.552254] LNetError: 113758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2558.584829] LNet: Removed LNI 192.168.204.42@tcp [ 2558.608681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2558.690604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2558.751524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2559.494375] Key type .llcrypt unregistered [ 2559.496193] Key type ._llcrypt unregistered [ 2572.451262] Key type ._llcrypt registered [ 2572.456572] Key type .llcrypt registered [ 2573.787581] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:52:10 (1786949530) [ 2574.222765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2574.250352] alg: No test for adler32 (adler32-zlib) [ 2575.181640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2575.256897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2575.277844] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2576.027955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2576.094457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2583.222456] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:52:20 (1786949540) [ 2583.500283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2583.541913] LNet: 115335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2583.559350] LNetError: 115335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2583.589583] LNet: Removed LNI 192.168.204.42@tcp [ 2583.643817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2583.693817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2583.738160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2590.763905] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:52:27 (1786949547) [ 2590.859229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2590.891329] LNet: 115908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2590.921505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2591.024915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2591.509569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2591.537776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2591.606239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2592.087156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2592.154572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2592.282441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2592.307326] LNet: 115949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2592.340392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2592.433216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2593.170266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2593.233489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2593.318555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2593.898771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2593.937857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2600.640143] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:52:37 (1786949557) [ 2600.968197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2600.992501] LNet: 116549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2601.008497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2601.079640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2601.110961] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2601.874677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2601.889773] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2602.782903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2602.879878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2603.240941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2603.273093] LNetError: 116829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2603.282740] LNet: Removed LNI 192.168.204.42@tcp [ 2603.313519] LNet: Removed LNI 192.168.204.42@tcp2 [ 2603.401534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2603.512316] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2603.525968] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2603.532584] LNet: 116835:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2604.334590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2604.615898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2604.682703] LNetError: 116872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2604.685276] LNetError: 116872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2604.698482] LNet: Removed LNI 192.168.204.42@tcp [ 2604.712384] LNet: Removed LNI 192.168.204.42@tcp2 [ 2604.732044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2604.778715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2604.818686] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2605.576099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2605.625353] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2606.801977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2606.851855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2606.919880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2607.133684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2607.155658] LNet: 117156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.160151] LNet: 117156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2607.165575] LNetError: 117156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2607.169684] LNetError: 117156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2608.223474] LNet: Removed LNI 192.168.204.42@tcp [ 2609.257413] LNet: Removed LNI 192.168.204.42@tcp2 [ 2609.293513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2609.376734] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2609.390023] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2609.396325] LNet: 117162:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2610.093770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2610.424714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2610.450979] LNet: Removed LNI 192.168.204.42@tcp [ 2611.509572] LNet: Removed LNI 192.168.204.42@tcp2 [ 2611.555148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2611.626702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2611.677277] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2612.601769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2612.633252] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2613.624597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2613.696446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2613.961036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2614.021763] LNetError: 117479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2614.031728] LNetError: 117479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2615.071196] LNet: Removed LNI 192.168.204.42@tcp [ 2615.080702] LNet: Removed LNI 192.168.204.42@tcp2 [ 2615.129906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2615.239951] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2615.246211] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2615.252568] LNet: 117485:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2616.279531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2616.566832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2616.598911] LNet: 117522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2616.614801] LNet: 117522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2617.699244] LNet: Removed LNI 192.168.204.42@tcp [ 2617.713905] LNet: Removed LNI 192.168.204.42@tcp2 [ 2617.758954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2617.880683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2617.932955] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2619.012457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2619.030506] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2620.344803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2620.395567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2620.450895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2620.630193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2621.663347] LNet: Removed LNI 192.168.204.42@tcp [ 2622.704635] LNet: Removed LNI 192.168.204.42@tcp2 [ 2622.765313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2622.844771] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2622.855701] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2622.860769] LNet: 117813:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2623.758501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2624.016322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2624.043796] LNetError: 117850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2624.049619] LNetError: 117850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2624.056515] LNet: Removed LNI 192.168.204.42@tcp [ 2625.132959] LNet: Removed LNI 192.168.204.42@tcp2 [ 2625.231189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2625.368241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2625.387721] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2626.475328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2626.521714] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2627.709277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2627.772016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2627.839733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2628.211791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2629.279260] LNet: Removed LNI 192.168.204.42@tcp [ 2629.295763] LNet: Removed LNI 192.168.204.42@tcp2 [ 2629.343424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2629.427719] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2629.435268] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2629.441389] LNet: 118153:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2630.093705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2630.408916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2631.455144] LNet: Removed LNI 192.168.204.42@tcp [ 2631.471158] LNet: Removed LNI 192.168.204.42@tcp2 [ 2631.507291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2631.592957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2631.611812] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2632.416691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2632.444449] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2633.430174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2633.498648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2633.533699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2633.604763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2634.089766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2634.122690] LNet: 118507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2634.129900] LNet: 118507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2635.167183] LNet: Removed LNI 192.168.204.42@tcp [ 2636.197778] LNet: Removed LNI 192.168.204.42@tcp2 [ 2636.261456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2636.343259] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2636.351699] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2637.035660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2645.988706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2648.066802] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:53:24 (1786949604) [ 2648.235067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2648.274578] LNetError: 119268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2648.279175] LNetError: 119268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2648.283205] LNet: Removed LNI 192.168.204.42@tcp [ 2649.319264] LNet: Removed LNI 192.168.204.42@tcp2 [ 2649.375212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2649.848454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2649.902257] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2650.784350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2650.809185] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2651.888264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2651.923475] LNet: Removed LNI 10.1.2.3@tcp [ 2651.941461] LNet: Removed LNI 192.168.204.42@tcp [ 2651.971040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2652.024314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2652.056358] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2652.062129] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2652.502047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2652.533472] LNet: Removed LNI 192.168.204.42@tcp [ 2652.549714] LNet: Removed LNI 10.1.2.3@tcp [ 2660.125361] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 02:53:36 (1786949616) [ 2660.469251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2660.566597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2660.640666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2660.690376] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2661.545349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2661.613077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2661.694263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2661.774800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2661.844993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2661.904501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2661.961989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2662.002457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2662.031439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2662.069617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2662.106510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2662.357825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2668.948422] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:53:46 (1786949626) [ 2669.436235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2669.481555] LNet: 120887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2669.494019] LNet: 120887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2670.571638] LNet: Removed LNI 192.168.204.42@tcp [ 2670.599670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2670.792124] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2671.033794] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2671.041339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2671.050549] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2673.621129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2673.680539] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2674.444221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2674.487905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2677.084984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2677.133918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2677.253744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.42@tcp -m PUT -e local_timeout -r 1 [ 2677.337959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2677.385940] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2677.390459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2677.623798] LNetError: 120953:0:(lib-move.c:4801:lnet_parse()) 192.168.204.42@tcp, src 192.168.204.42@tcp: Dropping PUT (error -108 looking up sender) [ 2678.432137] LNet: Removed LNI 10.1.2.3@tcp [ 2679.467449] LNet: Removed LNI 192.168.204.42@tcp [ 2679.937517] Key type .llcrypt unregistered [ 2679.944358] Key type ._llcrypt unregistered [ 2693.673844] Key type ._llcrypt registered [ 2693.676070] Key type .llcrypt registered [ 2695.234695] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:54:12 (1786949652) [ 2695.807835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2695.835497] alg: No test for adler32 (adler32-zlib) [ 2696.831477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2697.228276] LNet: 122269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2697.908490] Key type .llcrypt unregistered [ 2697.912833] Key type ._llcrypt unregistered [ 2704.499873] Key type ._llcrypt registered [ 2704.502637] Key type .llcrypt registered [ 2704.710038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2704.739152] alg: No test for adler32 (adler32-zlib) [ 2705.790837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2705.828286] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2712.925553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2713.210255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2713.926364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.142@tcp [ 2714.084568] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 2714.175159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2714.208185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2714.244618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2714.281540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2714.315979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2714.350632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2714.375742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2714.398482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2714.421307] LNet: 122722:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2714.436337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2714.462878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2714.490727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2714.515343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2714.546209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2714.570931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2715.254335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2715.476794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2715.517918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2715.550714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2715.576896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2715.614924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2715.648963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2715.693588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2715.753820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2715.777530] LNet: 122840:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2715.819956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2715.869692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2715.923208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2715.970602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2716.012718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2716.047321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2716.816666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.142@tcp [ 2717.115378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2717.161190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2717.219169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2717.275876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2717.317736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2717.366767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2717.423489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2717.491802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2717.523823] LNet: 122957:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2717.556877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2717.601462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2717.667492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2717.700713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2717.766302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2717.804392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2718.682557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2718.948993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2718.981579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2719.007918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2719.036551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2719.070868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2719.102413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2719.140757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2719.172787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2719.221944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2719.263956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2719.311806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2719.378796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2719.428821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2719.466730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2719.854503] LNet: 123124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2719.862110] LNetError: 123124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2719.916099] LNet: Removed LNI 192.168.204.42@tcp [ 2720.377645] Key type .llcrypt unregistered [ 2720.384641] Key type ._llcrypt unregistered [ 2746.009313] Key type ._llcrypt registered [ 2746.013312] Key type .llcrypt registered [ 2749.344988] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:55:04 (1786949704) [ 2750.205453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2750.233209] alg: No test for adler32 (adler32-zlib) [ 2751.188449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2751.639920] LNet: 124014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2753.035562] Key type .llcrypt unregistered [ 2753.041421] Key type ._llcrypt unregistered [ 2759.618461] Key type ._llcrypt registered [ 2759.626371] Key type .llcrypt registered [ 2760.015836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2760.062298] alg: No test for adler32 (adler32-zlib) [ 2761.298280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2761.349412] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2770.648146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2781.658514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2781.712789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2781.751350] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2783.329626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2784.126577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.142@tcp [ 2784.449858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2784.488728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2784.539432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2784.588721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2784.660066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2784.711646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2784.792318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2784.824694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2784.848739] LNet: 124671:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2784.876501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2784.913123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2784.929900] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 2784.970948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2785.015454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2785.061321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2785.107602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2786.087641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2786.434691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2786.505366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2786.556547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2786.624203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2786.704832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2786.768717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2786.844282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2786.882760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2786.904719] LNet: 124789:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2786.948993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2787.016684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2787.102625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2787.170979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2787.238436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2787.284609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2788.105455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.142@tcp [ 2788.355688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2788.386235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2788.415979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2788.452512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2788.493038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2788.538132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2788.578532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2788.624656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2788.672990] LNet: 124906:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2788.709712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2788.785420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2788.838940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2788.891266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2788.922726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2788.953410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2789.644284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2789.869869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2789.898623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2789.919614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2789.942270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2789.966133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2789.993639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2790.037634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2790.075434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2790.131755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2790.184587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2790.242630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2790.274449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2790.345224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2790.376471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2796.767550] LNet: 125108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2796.782497] LNetError: 125108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2796.788427] LNet: Removed LNI 192.168.204.42@tcp [ 2796.811156] LNet: Removed LNI 192.168.204.42@tcp1 [ 2797.442986] Key type .llcrypt unregistered [ 2797.450627] Key type ._llcrypt unregistered [ 2815.301384] Key type ._llcrypt registered [ 2815.303836] Key type .llcrypt registered [ 2817.127241] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 02:56:13 (1786949773) [ 2817.471183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2817.481609] alg: No test for adler32 (adler32-zlib) [ 2818.407123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2818.462909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2818.481634] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2819.479881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2819.515169] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2820.682760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2820.760653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2821.791213] LNet: Removed LNI 192.168.204.42@tcp [ 2830.075352] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 02:56:26 (1786949786) [ 2832.063794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2833.916420] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:56:31 (1786949791) [ 2842.567595] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:56:39 (1786949799) [ 2850.142701] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:56:47 (1786949807) [ 2859.061237] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:56:56 (1786949816) [ 2866.505454] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:57:03 (1786949823) [ 2874.703368] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:57:11 (1786949831) [ 2882.759654] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:57:19 (1786949839) [ 2890.865885] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:57:27 (1786949847) [ 2899.313921] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:57:36 (1786949856) [ 2906.646375] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:57:43 (1786949863) [ 2914.620524] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:57:51 (1786949871) [ 2922.390898] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:57:59 (1786949879) [ 2929.565418] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:58:06 (1786949886) [ 2937.685253] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:58:14 (1786949894) [ 2945.631298] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:58:22 (1786949902) [ 2953.581821] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:58:30 (1786949910) [ 2960.511451] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:58:37 (1786949917) [ 2969.076571] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:58:45 (1786949925) [ 2969.154294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2969.179429] LNet: 136803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2969.191770] LNetError: 136803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2969.207161] LNet: Removed LNI 192.168.204.42@tcp2 [ 2969.234974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2969.377291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2969.394987] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2970.522818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2970.566151] LNet: 136952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2970.577517] LNetError: 136952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2970.595992] LNet: Removed LNI 192.168.204.42@tcp [ 2970.632681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2970.734756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2970.781691] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2977.559888] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:58:54 (1786949934) [ 2984.971524] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:59:02 (1786949942) [ 2992.501482] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:59:09 (1786949949) [ 3000.918369] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:59:17 (1786949957) [ 3008.543590] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:59:25 (1786949965) [ 3015.956434] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:59:33 (1786949973) [ 3023.747373] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:59:40 (1786949980) [ 3032.625723] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:59:48 (1786949988) [ 3042.355542] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:59:59 (1786949999) [ 3051.347537] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 03:00:08 (1786950008) [ 3058.827738] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 03:00:16 (1786950016) [ 3065.066826] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 03:00:22 (1786950022) [ 3071.597895] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2724 sec ========== 03:00:28 (1786950028) [ 3071.982886] LNet: 144421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3072.006432] LNetError: 144421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3072.031342] LNet: Removed LNI 192.168.204.42@tcp [ 3072.611662] Key type .llcrypt unregistered [ 3072.614299] Key type ._llcrypt unregistered [ 3085.688111] Key type ._llcrypt registered [ 3085.691804] Key type .llcrypt registered [ 3086.146349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3086.166198] alg: No test for adler32 (adler32-zlib) [ 3087.294370] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 3087.613790] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3089.319171] Key type lgssc registered [ 3090.820222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3168.460468] Lustre: Mounted lustre-client - version 2.17.57_1_g72a0e6d [ 3175.455786] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3183.748591] Lustre: Unmounted lustre-client [ 3204.866306] Key type lgssc unregistered [ 3205.174249] LNet: 147395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3205.181952] LNetError: 147395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3206.253188] LNet: Removed LNI 192.168.204.42@tcp [ 3206.985294] Key type .llcrypt unregistered [ 3206.988364] Key type ._llcrypt unregistered