[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 427664850 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003302] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.007934] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009019] pid_max: default: 32768 minimum: 301 [ 0.010118] LSM: Security Framework initializing [ 0.011034] Yama: becoming mindful. [ 0.012023] SELinux: Initializing. [ 0.013046] *** VALIDATE selinux *** [ 0.021212] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026046] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027126] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028080] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029082] *** VALIDATE tmpfs *** [ 0.031403] *** VALIDATE proc *** [ 0.032185] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.035111] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036130] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038020] Spectre V2 : User space: Vulnerable [ 0.039015] Speculative Store Bypass: Vulnerable [ 0.042332] debug: unmapping init [mem 0xffffffff96859000-0xffffffff96860fff] [ 0.044153] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045586] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046015] ... version: 2 [ 0.047007] ... bit width: 48 [ 0.048006] ... generic registers: 4 [ 0.049006] ... value mask: 0000ffffffffffff [ 0.050006] ... max period: 00007fffffffffff [ 0.051007] ... fixed-purpose events: 3 [ 0.052007] ... event mask: 000000070000000f [ 0.054096] rcu: Hierarchical SRCU implementation. [ 0.056277] smp: Bringing up secondary CPUs ... [ 0.057452] x86: Booting SMP configuration: [ 0.058018] .... node #0, CPUs: #1 #2 #3 [ 0.061012] smp: Brought up 1 node, 4 CPUs [ 0.063010] smpboot: Max logical packages: 1 [ 0.064010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.192035] node 0 deferred pages initialised in 127ms [ 0.195016] devtmpfs: initialized [ 0.196401] x86/mm: Memory block size: 128MB [ 0.198824] gcov: version magic: 0x41383552 [ 0.202353] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.205124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.207448] pinctrl core: initialized pinctrl subsystem [ 0.208241] [ 0.208729] ************************************************************* [ 0.211018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.213012] ** ** [ 0.215017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.217013] ** ** [ 0.219016] ** This means that this kernel is built to expose internal ** [ 0.221014] ** IOMMU data structures, which may compromise security on ** [ 0.223016] ** your system. ** [ 0.224008] ** ** [ 0.226008] ** If you see this message and you are not debugging the ** [ 0.228016] ** kernel, report this immediately to your vendor! ** [ 0.230012] ** ** [ 0.232011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.234008] ************************************************************* [ 0.235672] NET: Registered protocol family 16 [ 0.237442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.239042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.241046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.244016] cpuidle: using governor menu [ 0.246110] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.249066] PCI: Using configuration type 1 for base access [ 0.250141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.258112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.259017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.260176] cryptd: max_cpu_qlen set to 1000 [ 0.261321] ACPI: Added _OSI(Module Device) [ 0.262000] ACPI: Added _OSI(Processor Device) [ 0.264009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.265023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.269436] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.273372] ACPI: Interpreter enabled [ 0.274000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.274000] ACPI: Using IOAPIC for interrupt routing [ 0.274091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.276329] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.286383] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.288031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.291031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.293075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.298123] acpiphp: Slot [2] registered [ 0.299153] acpiphp: Slot [3] registered [ 0.300076] acpiphp: Slot [4] registered [ 0.302067] acpiphp: Slot [5] registered [ 0.303106] acpiphp: Slot [6] registered [ 0.304078] acpiphp: Slot [7] registered [ 0.305046] acpiphp: Slot [8] registered [ 0.307141] acpiphp: Slot [9] registered [ 0.308101] acpiphp: Slot [10] registered [ 0.310111] acpiphp: Slot [11] registered [ 0.311149] acpiphp: Slot [12] registered [ 0.312068] acpiphp: Slot [13] registered [ 0.314090] acpiphp: Slot [14] registered [ 0.315109] acpiphp: Slot [15] registered [ 0.316069] acpiphp: Slot [16] registered [ 0.318091] acpiphp: Slot [17] registered [ 0.319048] acpiphp: Slot [18] registered [ 0.320143] acpiphp: Slot [19] registered [ 0.322116] acpiphp: Slot [20] registered [ 0.323101] acpiphp: Slot [21] registered [ 0.325198] acpiphp: Slot [22] registered [ 0.326052] acpiphp: Slot [23] registered [ 0.327139] acpiphp: Slot [24] registered [ 0.328065] acpiphp: Slot [25] registered [ 0.329094] acpiphp: Slot [26] registered [ 0.331123] acpiphp: Slot [27] registered [ 0.332127] acpiphp: Slot [28] registered [ 0.333126] acpiphp: Slot [29] registered [ 0.335151] acpiphp: Slot [30] registered [ 0.336085] acpiphp: Slot [31] registered [ 0.339128] PCI host bridge to bus 0000:00 [ 0.340019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.343030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.345021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349022] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.352032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354170] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.358351] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361754] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.367886] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.371547] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379469] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380740] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.383033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385687] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388620] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.397024] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.400011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405175] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.414027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.424015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438028] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.446366] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.452024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.458038] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.469032] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.476180] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.479397] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.481413] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.483399] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.485225] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.490611] iommu: Default domain type: Passthrough [ 0.491290] SCSI subsystem initialized [ 0.491950] ACPI: bus type USB registered [ 0.493062] usbcore: registered new interface driver usbfs [ 0.493969] usbcore: registered new interface driver hub [ 0.495054] usbcore: registered new device driver usb [ 0.495937] pps_core: LinuxPPS API ver. 1 registered [ 0.497006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.499027] PTP clock support registered [ 0.500097] EDAC MC: Ver: 3.0.0 [ 0.501408] PCI: Using ACPI for IRQ routing [ 0.503000] NetLabel: Initializing [ 0.504015] NetLabel: domain hash size = 128 [ 0.506018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.508124] NetLabel: unlabeled traffic allowed by default [ 0.511143] vgaarb: loaded [ 0.513324] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.514008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.520376] clocksource: Switched to clocksource kvm-clock [ 0.627513] VFS: Disk quotas dquot_6.6.0 [ 0.628980] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.631094] *** VALIDATE ramfs *** [ 0.632068] *** VALIDATE hugetlbfs *** [ 0.633283] pnp: PnP ACPI init [ 0.635248] pnp: PnP ACPI: found 6 devices [ 0.650799] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.652535] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.653708] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.654709] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.655959] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.657117] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.658379] NET: Registered protocol family 2 [ 0.660477] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.665163] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.669520] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.675849] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.680547] TCP: Hash tables configured (established 65536 bind 65536) [ 0.684761] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.690284] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.693516] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.696366] NET: Registered protocol family 1 [ 0.698980] RPC: Registered named UNIX socket transport module. [ 0.700615] RPC: Registered udp transport module. [ 0.702126] RPC: Registered tcp transport module. [ 0.703503] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.705204] NET: Registered protocol family 44 [ 0.706340] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.708297] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.710212] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.712062] PCI: CLS 0 bytes, default 64 [ 0.713623] Unpacking initramfs... [ 2.186171] debug: unmapping init [mem 0xffff8f08bcc64000-0xffff8f08bffcffff] [ 2.192041] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.194533] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.198862] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.768959] Initialise system trusted keyrings [ 2.771420] Key type blacklist registered [ 2.774609] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.784615] zbud: loaded [ 2.788496] *** VALIDATE nfs *** [ 2.789396] *** VALIDATE nfs4 *** [ 2.790853] pstore: using deflate compression [ 2.793846] Platform Keyring initialized [ 2.931431] NET: Registered protocol family 38 [ 2.936417] Key type asymmetric registered [ 2.938317] Asymmetric key parser 'x509' registered [ 2.940677] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.945179] io scheduler mq-deadline registered [ 2.948412] io scheduler kyber registered [ 2.951336] io scheduler bfq registered [ 2.953291] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.956440] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.958462] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.961676] ACPI: Power Button [PWRF] [ 3.070122] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.185503] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.294960] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.324253] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.354177] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.362814] Non-volatile memory driver v1.3 [ 3.365921] Linux agpgart interface v0.103 [ 3.400663] virtio_blk virtio1: [vda] 133160 512-byte logical blocks (68.2 MB/65.0 MiB) [ 3.403581] vda: detected capacity change from 0 to 68177920 [ 3.420217] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.423558] vdb: detected capacity change from 0 to 1073741824 [ 3.429716] libphy: Fixed MDIO Bus: probed [ 3.440091] usbcore: registered new interface driver usbserial_generic [ 3.442567] usbserial: USB Serial support registered for generic [ 3.445168] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.450596] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.452876] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.456508] mousedev: PS/2 mouse device common for all mice [ 3.459554] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.461326] rtc_cmos 00:05: RTC can wake from S4 [ 3.469460] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.469701] rtc_cmos 00:05: registered as rtc0 [ 3.476977] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.480937] intel_pstate: CPU model not supported [ 3.486488] hid: raw HID events driver (C) Jiri Kosina [ 3.487424] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.490508] usbcore: registered new interface driver usbhid [ 3.494452] usbhid: USB HID core driver [ 3.496655] drop_monitor: Initializing network drop monitor service [ 3.499275] Initializing XFRM netlink socket [ 3.502419] NET: Registered protocol family 10 [ 3.507735] Segment Routing with IPv6 [ 3.511371] NET: Registered protocol family 17 [ 3.516742] mpls_gso: MPLS GSO support [ 3.523873] RAS: Correctable Errors collector initialized. [ 3.526206] AVX version of gcm_enc/dec engaged. [ 3.528034] AES CTR mode by8 optimization enabled [ 3.630756] sched_clock: Marking stable (3630736138, 0)->(4331640874, -700904736) [ 3.633402] registered taskstats version 1 [ 3.635554] Loading compiled-in X.509 certificates [ 3.636916] zswap: loaded using pool lzo/zbud [ 3.672185] Key type big_key registered [ 3.712151] Key type encrypted registered [ 3.714307] ima: No TPM chip found, activating TPM-bypass! [ 3.716563] ima: Allocated hash algorithm: sha1 [ 3.718104] ima: No architecture policies found [ 3.720117] evm: Initialising EVM extended attributes: [ 3.722614] evm: security.selinux [ 3.723869] evm: security.ima [ 3.725100] evm: security.capability [ 3.726740] evm: HMAC attrs: 0x1 [ 3.731219] rtc_cmos 00:05: setting system clock to 2025-09-24 17:04:27 UTC (1758733467) [ 3.739313] debug: unmapping init [mem 0xffffffff97803000-0xffffffff979fffff] [ 3.743808] debug: unmapping init [mem 0xffffffff96582000-0xffffffff96858fff] [ 3.759986] Write protecting the kernel read-only data: 28672k [ 3.771535] debug: unmapping init [mem 0xffffffff94c03000-0xffffffff94dfffff] [ 3.780801] debug: unmapping init [mem 0xffffffff95514000-0xffffffff955fffff] [ 3.838864] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.857988] systemd[1]: Detected virtualization kvm. [ 3.860043] systemd[1]: Detected architecture x86-64. [ 3.861875] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.894192] systemd[1]: No hostname configured. [ 3.895846] systemd[1]: Set hostname to . [ 3.900357] random: systemd: uninitialized urandom read (16 bytes read) [ 3.905279] systemd[1]: Initializing machine ID from random generator. [ 3.976979] random: ln: uninitialized urandom read (6 bytes read) [ 4.096612] random: systemd: uninitialized urandom read (16 bytes read) [ 4.099512] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.112168] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 4.118499] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.189884] device-mapper: uevent: version 1.0.3 [ 5.193478] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.813456] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 7.442199] scsi host0: ata_piix [ 7.465726] scsi host1: ata_piix [ 7.477122] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.478796] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.447836] random: fast init done [ 13.248131] random: crng init done [ 13.251458] random: 7 urandom warning(s) missed due to ratelimiting [ 13.901154] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.761487] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.157023] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.527365] SELinux: Disabled at runtime. [ 18.598188] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.609355] systemd[1]: Detected virtualization kvm. [ 18.610712] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.541697] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.547375] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.555643] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.561316] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.563864] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.578377] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.590038] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ 19.705708] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.577581] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.315874] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.387820] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.248543] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.496637] EDAC sbridge: Ver: 1.1.2 [ 25.270470] Key type dns_resolver registered [ 26.017920] NFS: Registering the id_resolver key type [ 26.020449] Key type id_resolver registered [ 26.023381] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ 37.804499] hrtimer: interrupt took 4193854 ns Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg426-client login: [ 105.268351] libcfs: loading out-of-tree module taints kernel. [ 105.306231] Key type ._llcrypt registered [ 105.307794] Key type .llcrypt registered [ 105.619880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 105.635928] alg: No test for adler32 (adler32-zlib) [ 106.788583] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 107.257239] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 108.911214] Key type lgssc registered [ 109.856285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 181.322607] Lustre: Mounted lustre-client [ 184.368301] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 196.621176] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing check_logdir /tmp/testlogs/ [ 199.237798] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing yml_node [ 202.049227] Lustre: DEBUG MARKER: Client: 2.16.58.51 [ 203.492785] Lustre: DEBUG MARKER: MDS: 2.16.58.51 [ 204.956726] Lustre: DEBUG MARKER: OSS: 2.16.58.51 [ 205.736133] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 24 13:07:48 EDT 2025 [ 206.816940] Lustre: lustre-OST0000-osc-ffff8f091909c000: disconnect after 24s idle [ 213.465779] LustreError: 7913:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8f091909c000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 213.472247] LustreError: 7913:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 213.487087] Lustre: Unmounted lustre-client [ 224.076066] Key type lgssc unregistered [ 224.281689] LNet: 8358:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 224.288656] LNetError: 8358:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 224.303245] LNet: Removed LNI 192.168.204.26@tcp [ 224.710181] Key type .llcrypt unregistered [ 224.713427] Key type ._llcrypt unregistered [ 239.478107] Key type ._llcrypt registered [ 239.480720] Key type .llcrypt registered [ 239.649474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 239.675268] alg: No test for adler32 (adler32-zlib) [ 240.601817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 240.668813] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 240.687050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 243.408850] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 243.484251] LNet: 9224:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 243.487934] LNetError: 9224:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 243.497460] LNet: Removed LNI 192.168.204.26@tcp [ 243.973098] Key type .llcrypt unregistered [ 243.974955] Key type ._llcrypt unregistered [ 247.825635] Key type ._llcrypt registered [ 247.828981] Key type .llcrypt registered [ 248.779240] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 13:08:31 (1758733711) [ 248.993564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 249.014869] alg: No test for adler32 (adler32-zlib) [ 249.910317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.971394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.985556] LNet: 9636:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 250.000075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.038974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 250.049363] LNet: 9643:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 254.192835] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 13:08:37 (1758733717) [ 254.255974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.302078] LNet: 10212:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 254.318554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.398597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 258.747420] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 13:08:41 (1758733721) [ 258.879362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.903091] LNet: 10805:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.921949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.988399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 263.139844] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 13:08:46 (1758733726) [ 263.231182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 263.249967] LNet: 11388:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 263.277611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 263.323891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 267.259969] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 13:08:50 (1758733730) [ 267.432692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.443949] LNet: 11973:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 267.453851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.533260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 267.832901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 272.005466] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 13:08:54 (1758733734) [ 272.052977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.084907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.152477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 272.421251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 272.784227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 277.098531] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 13:08:59 (1758733739) [ 277.146351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 277.159472] LNet: 13149:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 277.172537] LNet: 13149:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 277.207298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 277.260427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 281.299753] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 13:09:04 (1758733744) [ 281.383655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 281.432083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 281.475663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 281.503321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 281.900064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 281.935741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 282.254185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 282.285369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 282.598431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 282.619932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 282.856788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 282.882907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 283.157592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 283.186753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 283.425640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 283.445255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 283.714216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 283.738827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 283.981740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 284.007987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 284.248192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 284.269252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 284.516380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 284.542735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 284.809626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 284.830903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 285.048269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 285.077437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 289.280525] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 13:09:12 (1758733752) [ 289.341661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.366572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.419228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 289.438510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 293.465600] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 13:09:16 (1758733756) [ 293.544479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.560836] LNet: 15005:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.569722] LNet: 15005:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 293.585210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.645431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 293.668485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 297.848638] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 13:09:20 (1758733760) [ 298.027578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.058789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.112749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 298.145759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 301.810933] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 13:09:24 (1758733764) [ 301.918585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.947196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.999246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 302.029350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 305.151914] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 13:09:28 (1758733768) [ 305.211474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.242529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.303438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 305.333279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 308.818893] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 13:09:31 (1758733771) [ 308.869061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.898273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.941923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 308.961801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 312.978674] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 13:09:35 (1758733775) [ 313.052356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.082925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.147580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 313.420771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 317.579825] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 13:09:40 (1758733780) [ 317.698540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.730947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.867353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 318.027386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 322.520342] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 13:09:45 (1758733785) [ 322.575966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.615738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.705657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 322.864793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 327.543802] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 13:09:50 (1758733790) [ 327.606931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.620400] LNet: 19759:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.623238] LNet: 19759:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 327.635715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.922727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 328.115022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 332.654938] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 13:09:55 (1758733795) [ 332.717631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.746407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.809529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 332.843887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 336.999744] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 13:09:59 (1758733799) [ 337.129044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.153451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.203223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 337.244300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 341.564639] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 13:10:04 (1758733804) [ 341.636563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.694802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.742272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 346.246322] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 13:10:09 (1758733809) [ 346.333056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.387387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.445427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 350.148026] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 13:10:13 (1758733813) [ 350.256832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.277834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.318631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 350.577287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 354.724720] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 13:10:17 (1758733817) [ 354.770905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.812432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.883965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 354.909951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 359.128602] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 13:10:21 (1758733821) [ 359.192856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.235475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.314815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 359.342233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 363.425541] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 13:10:26 (1758733826) [ 363.493104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.531871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.595326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 363.631587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 368.151952] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 13:10:30 (1758733830) [ 368.268801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.301708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.373158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 368.403273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 372.846370] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 13:10:35 (1758733835) [ 372.976246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.008835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.067480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 373.126510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 373.142644] LNetError: 25737:0:(peer.c:2209:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 373.180885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 373.210076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 377.488258] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 13:10:40 (1758733840) [ 377.616233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.717068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.761780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.SSYFlzmb6Z [ 377.818947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.SSYFlzmb6Z [ 377.864534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.SSYFlzmb6Z [ 381.665330] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 13:10:44 (1758733844) [ 381.744350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.767597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.806951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 381.830773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 381.863949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 385.933042] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 13:10:48 (1758733848) [ 385.995762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.039927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.101394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 387.673792] LNet: Message forwarding will be enabled in 2s [ 390.751163] LNet: Message forwarding enabled [ 390.936182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 390.949816] LNet: Message forwarding will stop in 2s to 6s [ 395.219440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 395.627470] LNet: Message forwarding disabled [ 396.461394] LNet: Message forwarding will be enabled in 2s [ 399.711560] LNet: Message forwarding enabled [ 400.270634] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 13:11:03 (1758733863) [ 400.316040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.328945] LNet: 28173:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.331892] LNet: 28173:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 401.023793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.070401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 404.905223] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 13:11:07 (1758733867) [ 404.991702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.013661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.058421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 405.079669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 405.100095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 405.123880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 405.140778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 405.159240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 405.176318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 405.201829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 405.227389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 405.251882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 405.277405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 405.304762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 405.330455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 405.355965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 405.381699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 405.403656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 405.432860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 405.452327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 405.473656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 405.486863] LNetError: 28804:0:(router.c:628:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 409.008615] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 13:11:11 (1758733871) [ 409.082361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.122820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.220284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 413.216723] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 13:11:16 (1758733876) [ 414.401746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg426-server [ 415.417503] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 13:11:18 (1758733878) [ 415.485150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.515343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.566138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 415.581767] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 416.605167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[27-31]@tcp [ 416.618828] LNet: 30516:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 416.642601] LNetError: 30308:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.30@tcp is being used as a gateway but routing feature is not turned on [ 416.651830] LNetError: 30308:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.30@tcp has gone from up to down [ 417.638019] LNetError: 30308:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.29@tcp is being used as a gateway but routing feature is not turned on [ 417.647712] LNetError: 30308:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 418.664637] LNetError: 30308:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.29@tcp is being used as a gateway but routing feature is not turned on [ 418.671169] LNetError: 30308:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 419.679740] LNetError: 30313:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.27@tcp has gone from up to down [ 419.690025] LNetError: 30313:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 420.615240] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 13:11:23 (1758733883) [ 420.709956] LNetError: 30308:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.30@tcp is being used as a gateway but routing feature is not turned on [ 420.718632] LNetError: 30308:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 421.604942] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg426-server [ 422.425212] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 13:11:25 (1758733885) [ 422.462741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.753714] LNetError: 30307:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.204.28@tcp [ 423.520875] LNetError: 31444:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 423.543817] LNet: Removed LNI 192.168.204.26@tcp [ 423.558715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.601077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 423.615461] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 423.988691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[27-32/2]@tcp [ 424.005564] LNet: 31565:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 424.010564] LNet: 31565:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 424.022027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[27-32/2]@tcp [ 424.027285] LNetError: 31455:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.29@tcp has gone from up to down [ 427.921275] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 13:11:30 (1758733890) [ 428.003569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.015465] LNetError: 32155:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 428.026410] LNet: Removed LNI 192.168.204.26@tcp [ 428.040628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.081676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 428.108346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.128589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.167252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.198882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.217614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.247633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 428.293131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.317217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.368406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.432754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.457918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.490979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 428.541499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.568151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.606756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.656938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.680972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.714464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 428.764813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.798414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.851396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.908905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.937612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.969789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 429.019336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.039342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.075149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 429.130237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.151849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.193284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 429.235754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.268636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.313295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.783882] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 13:11:35 (1758733895) [ 433.952743] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg426-server [ 435.058292] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 13:11:37 (1758733897) [ 436.242830] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg426-server [ 437.194605] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 13:11:40 (1758733900) [ 437.633399] Key type .llcrypt unregistered [ 437.634900] Key type ._llcrypt unregistered [ 440.709870] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 440.825383] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 440.827371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 440.830716] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 443.090086] Key type ._llcrypt registered [ 443.091669] Key type .llcrypt registered [ 443.191207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 443.200712] alg: No test for adler32 (adler32-zlib) [ 444.040405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.069497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 444.106415] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 444.437904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 444.448620] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 445.368478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 445.379438] LNet: Removed LNI 10.1.2.3@tcp [ 445.423604] LNet: 34267:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.427601] LNetError: 34267:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 445.437641] LNet: Removed LNI 192.168.204.26@tcp [ 445.724581] Key type .llcrypt unregistered [ 445.726399] Key type ._llcrypt unregistered [ 450.910273] Key type ._llcrypt registered [ 450.913545] Key type .llcrypt registered [ 451.636732] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 13:11:54 (1758733914) [ 451.956421] Key type .llcrypt unregistered [ 451.958362] Key type ._llcrypt unregistered [ 454.918179] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 455.043560] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 455.046514] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 455.050453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 457.333117] Key type ._llcrypt registered [ 457.334778] Key type .llcrypt registered [ 457.449680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 457.457288] alg: No test for adler32 (adler32-zlib) [ 458.299437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.337990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 458.353737] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 458.777788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 458.788983] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 460.410432] LNet: 35747:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.414830] LNetError: 35747:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 460.419268] LNet: Removed LNI 192.168.204.26@tcp [ 460.430724] LNet: Removed LNI 10.1.2.3@tcp [ 460.801738] Key type .llcrypt unregistered [ 460.803656] Key type ._llcrypt unregistered [ 467.200728] Key type ._llcrypt registered [ 467.204657] Key type .llcrypt registered [ 468.102501] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 13:12:10 (1758733930) [ 468.486336] Key type .llcrypt unregistered [ 468.487887] Key type ._llcrypt unregistered [ 471.454335] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 471.563763] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 471.572583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 471.575382] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 473.866532] Key type ._llcrypt registered [ 473.871180] Key type .llcrypt registered [ 474.016339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 474.027802] alg: No test for adler32 (adler32-zlib) [ 474.872854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.915394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 474.931216] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 476.083474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 476.105130] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 477.434243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 477.451824] LNet: Removed LNI 10.1.2.3@tcp [ 477.463712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 477.485815] LNet: Removed LNI 10.1.2.31@tcp [ 477.495488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 477.520475] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 478.029570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 478.044577] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 478.636926] LNet: 37644:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 478.642913] LNetError: 37644:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 478.648229] LNet: Removed LNI 10.1.2.3@tcp [ 478.655536] LNet: Removed LNI 10.1.2.31@tcp [ 479.022649] Key type .llcrypt unregistered [ 479.024415] Key type ._llcrypt unregistered [ 485.449985] Key type ._llcrypt registered [ 485.451384] Key type .llcrypt registered [ 486.256802] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 13:12:29 (1758733949) [ 486.504858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 486.513384] alg: No test for adler32 (adler32-zlib) [ 487.386927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.422212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 487.441784] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 487.982159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.000227] LNet: 38588:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 488.007124] LNetError: 38588:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 488.020510] LNet: Removed LNI 192.168.204.26@tcp [ 488.046759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.122945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.26@tcp [ 488.169667] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 488.687642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.26@tcp [ 488.713421] LNet: Removed LNI 192.168.204.26@tcp [ 488.772972] LNet: 38720:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.194512] Key type .llcrypt unregistered [ 489.196073] Key type ._llcrypt unregistered [ 495.734180] Key type ._llcrypt registered [ 495.735773] Key type .llcrypt registered [ 496.558289] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 13:12:39 (1758733959) [ 497.650755] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg426-server [ 498.540204] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 13:12:41 (1758733961) [ 498.765360] Key type .llcrypt unregistered [ 498.767088] Key type ._llcrypt unregistered [ 501.522729] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 501.616873] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 501.619193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 501.622313] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 503.873870] Key type ._llcrypt registered [ 503.875347] Key type .llcrypt registered [ 503.990440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 504.001348] alg: No test for adler32 (adler32-zlib) [ 504.827810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.872532] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 505.250071] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 506.657332] LNet: 40545:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 506.663359] LNetError: 40545:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 506.666703] LNet: Removed LNI 192.168.204.26@tcp1 [ 506.675085] LNet: Removed LNI 10.1.2.3@tcp2 [ 507.098709] Key type .llcrypt unregistered [ 507.100502] Key type ._llcrypt unregistered [ 513.995304] Key type ._llcrypt registered [ 513.997023] Key type .llcrypt registered [ 514.886967] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 13:12:57 (1758733977) [ 515.129452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 515.138914] alg: No test for adler32 (adler32-zlib) [ 516.002949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.050826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 516.068769] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 516.637985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.712035] LNet: 41502:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 516.720061] LNetError: 41502:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 516.735546] LNet: Removed LNI 192.168.204.26@tcp [ 516.760728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.826621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 516.848104] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 520.961782] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 13:13:03 (1758733983) [ 521.027306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.042344] LNet: 42091:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.045729] LNetError: 42091:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.059395] LNet: Removed LNI 192.168.204.26@tcp [ 521.071991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.112913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 521.134114] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 521.695124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.26 [ 521.708723] LNet: Removed LNI 192.168.204.26@tcp [ 525.483323] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 13:13:08 (1758733988) [ 525.562386] LNet: 42791:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 525.572704] LNetError: 42791:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 525.587314] LNet: Removed LNI 192.168.204.26@tcp [ 526.070714] Key type .llcrypt unregistered [ 526.072368] Key type ._llcrypt unregistered [ 528.721458] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 528.852621] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 528.854845] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 528.857698] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 531.256658] Key type ._llcrypt registered [ 531.258095] Key type .llcrypt registered [ 531.341899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 531.353504] alg: No test for adler32 (adler32-zlib) [ 532.406305] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 535.523745] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 13:13:18 (1758733998) [ 535.576625] LNet: 43693:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.580722] LNetError: 43693:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 535.592259] LNet: Removed LNI 10.1.2.3@tcp [ 535.920125] Key type .llcrypt unregistered [ 535.921863] Key type ._llcrypt unregistered [ 538.324297] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 538.437728] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 538.440189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 538.443833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 540.864806] Key type ._llcrypt registered [ 540.866619] Key type .llcrypt registered [ 540.952766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 540.964491] alg: No test for adler32 (adler32-zlib) [ 541.805973] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 544.856849] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 13:13:27 (1758734007) [ 544.963664] LNet: 44584:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 544.967764] LNetError: 44584:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 544.977309] LNet: Removed LNI 10.1.2.3@tcp [ 545.308528] Key type .llcrypt unregistered [ 545.311072] Key type ._llcrypt unregistered [ 548.211910] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 548.331807] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 548.335154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 548.339384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 550.684653] Key type ._llcrypt registered [ 550.686036] Key type .llcrypt registered [ 550.755339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 550.762443] alg: No test for adler32 (adler32-zlib) [ 551.601046] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 554.674565] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 13:13:37 (1758734017) [ 554.736393] LNet: 45467:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.739836] LNetError: 45467:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 554.749720] LNet: Removed LNI 10.1.2.3@tcp [ 555.068604] Key type .llcrypt unregistered [ 555.069974] Key type ._llcrypt unregistered [ 557.448563] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 557.558129] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 557.560912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 557.564587] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 559.928705] Key type ._llcrypt registered [ 559.931195] Key type .llcrypt registered [ 560.043416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 560.053126] alg: No test for adler32 (adler32-zlib) [ 560.932422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.981411] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 565.225364] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 13:13:48 (1758734028) [ 565.463348] LNet: 46693:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.466103] LNetError: 46693:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 565.480554] LNet: Removed LNI 10.1.2.3@tcp [ 565.842951] Key type .llcrypt unregistered [ 565.844885] Key type ._llcrypt unregistered [ 568.251684] Key type ._llcrypt registered [ 568.253294] Key type .llcrypt registered [ 568.340567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 568.348454] alg: No test for adler32 (adler32-zlib) [ 569.177946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 569.195337] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 572.995913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 573.268608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_interrupt -r 1 [ 573.288068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_interrupt -r 1 [ 573.307059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 573.318416] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 573.456425] LNet: 47118:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 573.490483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 573.509227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 573.528767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 573.767796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_dropped -r 1 [ 573.788550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_dropped -r 1 [ 573.809960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 573.820791] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 573.973353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 573.990572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 574.009948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 574.233292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_aborted -r 1 [ 574.252380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_aborted -r 1 [ 574.273170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 574.418036] LNet: 47312:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 574.421042] LNet: 47312:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 574.463955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 574.482722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 574.501565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 574.750926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_no_route -r 1 [ 574.770265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_no_route -r 1 [ 574.789724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 574.977367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 574.995435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 575.014400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 575.241906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_timeout -r 1 [ 575.259444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_timeout -r 1 [ 575.280228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 575.291383] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 575.294683] LNet: Skipped 2 previous similar messages [ 575.430940] LNet: 47506:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 575.434703] LNet: 47506:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 575.470661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 575.488341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 575.506349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 575.733097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_error -r 1 [ 575.751785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_error -r 1 [ 575.770876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 575.958784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 575.976368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 575.994919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 576.115767] LNet: 47639:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.119126] LNetError: 47639:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.131531] LNet: Removed LNI 192.168.204.26@tcp [ 576.327499] Key type .llcrypt unregistered [ 576.329258] Key type ._llcrypt unregistered [ 583.075447] Key type ._llcrypt registered [ 583.076546] Key type .llcrypt registered [ 583.623291] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 13:14:06 (1758734046) [ 583.995551] Key type .llcrypt unregistered [ 583.997280] Key type ._llcrypt unregistered [ 586.327041] Key type ._llcrypt registered [ 586.328937] Key type .llcrypt registered [ 586.438817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 586.449099] alg: No test for adler32 (adler32-zlib) [ 587.276301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 587.291361] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 590.896894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 595.430102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.447840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 595.461071] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 596.166680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_interrupt -r 1 [ 596.185860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_interrupt -r 1 [ 596.205082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_interrupt -r 1 [ 596.223549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_interrupt -r 1 [ 596.243524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_interrupt -r 1 [ 596.262861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_interrupt -r 1 [ 596.281484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_interrupt -r 1 [ 596.303288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_interrupt -r 1 [ 596.326753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 596.338918] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 596.342512] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 596.504236] LNet: 49084:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 596.550908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 596.575470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 596.602897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 596.828363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_dropped -r 1 [ 596.848678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_dropped -r 1 [ 596.868283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_dropped -r 1 [ 596.889662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_dropped -r 1 [ 596.909720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_dropped -r 1 [ 596.929366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_dropped -r 1 [ 596.948623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_dropped -r 1 [ 596.970564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_dropped -r 1 [ 596.990856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 597.003939] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 597.007675] LNet: Skipped 2 previous similar messages [ 597.162468] LNet: 49193:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 597.197210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 597.213840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 597.238199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 597.475790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_aborted -r 1 [ 597.496941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_aborted -r 1 [ 597.518936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_aborted -r 1 [ 597.541329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_aborted -r 1 [ 597.560369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_aborted -r 1 [ 597.582684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_aborted -r 1 [ 597.608583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_aborted -r 1 [ 597.631167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_aborted -r 1 [ 597.653421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 597.848289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 597.867330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 597.887168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 598.138610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_no_route -r 1 [ 598.158124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_no_route -r 1 [ 598.175937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_no_route -r 1 [ 598.197926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_no_route -r 1 [ 598.218312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_no_route -r 1 [ 598.236475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_no_route -r 1 [ 598.262700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_no_route -r 1 [ 598.281382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_no_route -r 1 [ 598.304479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 598.318842] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 598.322223] LNet: Skipped 7 previous similar messages [ 598.471082] LNet: 49412:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 598.474268] LNet: 49412:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 598.509893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 598.526618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 598.544263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 598.807976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_timeout -r 1 [ 598.831394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_timeout -r 1 [ 598.854423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_timeout -r 1 [ 598.875644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_timeout -r 1 [ 598.896961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_timeout -r 1 [ 598.916288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_timeout -r 1 [ 598.936297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_timeout -r 1 [ 598.956678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_timeout -r 1 [ 599.001157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 599.188275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 599.206224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 599.226683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 599.493973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_error -r 1 [ 599.513257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_error -r 1 [ 599.532732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_error -r 1 [ 599.554174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_error -r 1 [ 599.582985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_error -r 1 [ 599.603220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_error -r 1 [ 599.625370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_error -r 1 [ 599.648313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_error -r 1 [ 599.673194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 599.867599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 599.890697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 599.913808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 602.423893] LNet: 49700:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.427386] LNetError: 49700:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.433091] LNet: Removed LNI 192.168.204.26@tcp [ 603.493874] LNet: Removed LNI 192.168.204.26@tcp1 [ 603.709947] Key type .llcrypt unregistered [ 603.712147] Key type ._llcrypt unregistered [ 610.966871] Key type ._llcrypt registered [ 610.968654] Key type .llcrypt registered [ 611.655764] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 13:14:34 (1758734074) [ 612.025524] Key type .llcrypt unregistered [ 612.027268] Key type ._llcrypt unregistered [ 614.327591] Key type ._llcrypt registered [ 614.329020] Key type .llcrypt registered [ 614.403149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 614.410415] alg: No test for adler32 (adler32-zlib) [ 615.234865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 615.250820] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 618.661968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 618.925302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_dropped -r 1 [ 618.944474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_dropped -r 1 [ 618.963106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 618.974327] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 619.106200] LNet: 50932:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 619.139258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 619.156938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 619.174650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 619.381794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_error -r 1 [ 619.401235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_error -r 1 [ 619.419992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 619.586052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 619.604731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 619.621903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 619.843810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_timeout -r 1 [ 619.864592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_timeout -r 1 [ 619.882251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 619.893329] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 619.896554] LNet: Skipped 1 previous similar message [ 620.024421] LNet: 51126:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 620.028857] LNet: 51126:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 620.070108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 620.091788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 620.110305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 620.230637] LNet: 51162:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.235593] LNetError: 51162:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.243364] LNet: Removed LNI 192.168.204.26@tcp [ 620.419903] Key type .llcrypt unregistered [ 620.421721] Key type ._llcrypt unregistered [ 626.914143] Key type ._llcrypt registered [ 626.915560] Key type .llcrypt registered [ 627.507615] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 13:14:50 (1758734090) [ 627.908205] Key type .llcrypt unregistered [ 627.910176] Key type ._llcrypt unregistered [ 630.704459] Key type ._llcrypt registered [ 630.705844] Key type .llcrypt registered [ 630.836723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 630.846876] alg: No test for adler32 (adler32-zlib) [ 631.671581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 631.692033] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 635.314852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 638.770901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.806364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 638.819356] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 639.551332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_dropped -r 1 [ 639.569955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e remote_dropped -r 1 [ 639.591193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_dropped -r 1 [ 639.613246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e remote_dropped -r 1 [ 639.631488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e remote_dropped -r 1 [ 639.655719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e remote_dropped -r 1 [ 639.675985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e remote_dropped -r 1 [ 639.698667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e remote_dropped -r 1 [ 639.717184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 639.729616] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 639.733362] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.126@tcp [ 639.869587] LNet: 52607:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 639.906304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 639.925037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 639.941324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 639.959768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 640.181395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_error -r 1 [ 640.198577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e remote_error -r 1 [ 640.220136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_error -r 1 [ 640.237926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e remote_error -r 1 [ 640.256913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e remote_error -r 1 [ 640.274180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e remote_error -r 1 [ 640.300108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e remote_error -r 1 [ 640.319966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e remote_error -r 1 [ 640.341451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 640.355878] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 640.358931] LNet: Skipped 2 previous similar messages [ 640.499366] LNet: 52718:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 640.533386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 640.549571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 640.566733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 640.584393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 640.817923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_timeout -r 1 [ 640.839292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e remote_timeout -r 1 [ 640.858391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_timeout -r 1 [ 640.878433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e remote_timeout -r 1 [ 640.896972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e remote_timeout -r 1 [ 640.915528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e remote_timeout -r 1 [ 640.934239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e remote_timeout -r 1 [ 640.953870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e remote_timeout -r 1 [ 640.975916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 641.168973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 641.186061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 641.204126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 641.224589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 643.808680] LNet: 52901:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.812759] LNetError: 52901:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.817660] LNet: Removed LNI 192.168.204.26@tcp [ 643.825516] LNet: Removed LNI 192.168.204.26@tcp1 [ 644.036488] Key type .llcrypt unregistered [ 644.038495] Key type ._llcrypt unregistered [ 651.592070] Key type ._llcrypt registered [ 651.593506] Key type .llcrypt registered [ 652.297655] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 13:15:15 (1758734115) [ 652.536537] Key type .llcrypt unregistered [ 652.538858] Key type ._llcrypt unregistered [ 654.972427] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 655.031112] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 655.034959] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 655.037433] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 657.316499] Key type ._llcrypt registered [ 657.317849] Key type .llcrypt registered [ 657.389395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 657.398261] alg: No test for adler32 (adler32-zlib) [ 658.239362] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 658.263983] LNet: 53986:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.268870] LNetError: 53986:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.281578] LNet: Removed LNI 192.168.204.26@tcp [ 658.611501] Key type .llcrypt unregistered [ 658.613127] Key type ._llcrypt unregistered [ 661.011867] Key type ._llcrypt registered [ 661.013559] Key type .llcrypt registered [ 661.081836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 661.090766] alg: No test for adler32 (adler32-zlib) [ 661.912562] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 661.914711] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 661.932769] LNet: 54255:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.936362] LNetError: 54255:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.939489] LNet: Removed LNI 192.168.204.26@tcp [ 661.952359] LNet: Removed LNI 10.1.2.3@tcp1 [ 662.259511] Key type .llcrypt unregistered [ 662.260824] Key type ._llcrypt unregistered [ 664.642326] Key type ._llcrypt registered [ 664.644096] Key type .llcrypt registered [ 664.724659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 664.732420] alg: No test for adler32 (adler32-zlib) [ 665.550067] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 665.552379] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 665.570251] LNet: 54530:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.573645] LNetError: 54530:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.576790] LNet: Removed LNI 192.168.204.26@tcp [ 665.583532] LNet: Removed LNI 10.1.2.3@tcp [ 665.874425] Key type .llcrypt unregistered [ 665.876310] Key type ._llcrypt unregistered [ 668.383749] Key type ._llcrypt registered [ 668.385295] Key type .llcrypt registered [ 668.475065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 668.485587] alg: No test for adler32 (adler32-zlib) [ 669.308676] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 669.311781] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 669.330093] LNet: 54799:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.332992] LNetError: 54799:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.335766] LNet: Removed LNI 192.168.204.26@tcp [ 669.343134] LNet: Removed LNI 10.1.2.3@tcp [ 669.640688] Key type .llcrypt unregistered [ 669.642272] Key type ._llcrypt unregistered [ 671.971975] Key type ._llcrypt registered [ 671.973330] Key type .llcrypt registered [ 672.060656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 672.066551] alg: No test for adler32 (adler32-zlib) [ 672.885613] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 672.888606] LNetError: 55059:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 672.893456] LNetError: Error -100 starting up LNI tcp [ 672.901504] LNet: Removed LNI 192.168.204.26@tcp [ 673.189393] Key type .llcrypt unregistered [ 673.191034] Key type ._llcrypt unregistered [ 677.637637] Key type ._llcrypt registered [ 677.639019] Key type .llcrypt registered [ 678.234962] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 13:15:41 (1758734141) [ 678.599345] Key type .llcrypt unregistered [ 678.601066] Key type ._llcrypt unregistered [ 680.988245] Key type ._llcrypt registered [ 680.989893] Key type .llcrypt registered [ 681.087662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 681.096573] alg: No test for adler32 (adler32-zlib) [ 681.954031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 681.970236] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 685.540603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 685.858837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e network_timeout -r 1 [ 685.879140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e network_timeout -r 1 [ 685.898175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 685.908455] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 686.055878] LNet: 56299:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 686.089212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 686.106340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 686.220841] LNet: 56333:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.224340] LNetError: 56333:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.238979] LNet: Removed LNI 192.168.204.26@tcp [ 686.430596] Key type .llcrypt unregistered [ 686.432410] Key type ._llcrypt unregistered [ 694.345115] Key type ._llcrypt registered [ 694.347849] Key type .llcrypt registered [ 694.504404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 694.515373] alg: No test for adler32 (adler32-zlib) [ 695.347842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 695.375834] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 699.335718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 703.739422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.762883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 703.777197] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 704.687949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e network_timeout -r 1 [ 704.708214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e network_timeout -r 1 [ 704.730776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e network_timeout -r 1 [ 704.763232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e network_timeout -r 1 [ 704.790450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e network_timeout -r 1 [ 704.812657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e network_timeout -r 1 [ 704.841904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e network_timeout -r 1 [ 704.870509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e network_timeout -r 1 [ 704.897942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 704.919357] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 705.111131] LNet: 57191:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 705.166585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 705.192324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 707.946232] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 708.152072] LNet: 57260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.159154] LNetError: 57260:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.165339] LNet: Removed LNI 192.168.204.26@tcp [ 708.174341] LNet: Removed LNI 192.168.204.26@tcp1 [ 708.446580] Key type .llcrypt unregistered [ 708.448136] Key type ._llcrypt unregistered [ 716.667151] Key type ._llcrypt registered [ 716.669422] Key type .llcrypt registered [ 717.497781] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 13:16:20 (1758734180) [ 717.711648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 717.723482] alg: No test for adler32 (adler32-zlib) [ 718.563048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.595553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.610371] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 718.999821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 719.017427] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 719.507810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 719.543752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 719.574291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 719.600231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 719.624636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 719.656613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 719.678823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.26@tcp [ 719.696860] LNetError: 58349:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.204.26@tcp to recovery [ 720.607339] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 720.607444] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 722.655600] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 722.933309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 726.752396] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 727.235979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 727.254168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 727.286441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.303380] LNet: 58466:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.307267] LNetError: 58466:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.309721] LNet: Removed LNI 192.168.204.26@tcp [ 727.325455] LNet: Removed LNI 192.168.204.26@tcp1 [ 727.335720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.366760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.384707] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 727.763027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 727.777172] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 728.241502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 728.262336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 728.298438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 728.325784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 728.350608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 728.375260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 728.395529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 728.414952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.26@tcp [ 728.427246] LNetError: 58732:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.204.26@tcp to recovery [ 729.380091] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 731.728650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 735.519334] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 735.522644] LNet: Skipped 1 previous similar message [ 740.329673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 740.355782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 740.378500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 744.031634] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 13:16:47 (1758734207) [ 744.124915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.143966] LNet: 59457:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.152129] LNetError: 59457:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.155528] LNetError: 59457:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 744.159556] LNet: Removed LNI 192.168.204.26@tcp [ 744.172915] LNet: Removed LNI 192.168.204.26@tcp1 [ 744.190661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.223727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 744.239896] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 744.621325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 744.632085] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 745.096077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 745.132605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 745.152065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 745.172634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 745.193863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 745.215212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 745.235389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.26@tcp --health 0 [ 745.286819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 746.271428] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.26@tcp [ 746.271850] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 746.277158] LNet: Skipped 1 previous similar message [ 761.027497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 761.096699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 761.118328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 761.137494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.26@tcp --health 500 [ 761.199168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 768.564826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 768.597733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.609314] LNet: 59961:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.622216] LNetError: 59961:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.625774] LNetError: 59961:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 768.629554] LNet: Removed LNI 192.168.204.26@tcp [ 768.636382] LNet: Removed LNI 192.168.204.26@tcp1 [ 768.647882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.675072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 768.689093] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 769.018042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 769.028507] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 769.413081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 769.434785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 769.471438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 769.489312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 769.507411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 769.527512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 769.545392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 769.563755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.26@tcp --health 0 [ 770.720021] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 770.722952] LNet: Skipped 3 previous similar messages [ 771.751887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 781.253245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 781.268398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 781.284148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 784.146568] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:17:27 (1758734247) [ 784.359041] LNet: 61011:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.362607] LNetError: 61011:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.367515] LNetError: 61011:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 784.372727] LNet: Removed LNI 192.168.204.26@tcp [ 784.381601] LNet: Removed LNI 192.168.204.26@tcp1 [ 784.685773] Key type .llcrypt unregistered [ 784.687353] Key type ._llcrypt unregistered [ 787.115692] Key type ._llcrypt registered [ 787.117473] Key type .llcrypt registered [ 787.206958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 787.215632] alg: No test for adler32 (adler32-zlib) [ 788.048515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 788.067813] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 792.684716] Lustre: DEBUG MARKER: Initial discovery [ 792.732530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.126@tcp [ 793.999768] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 794.063710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 794.090247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.126@tcp [ 794.105880] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 794.119725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 794.164317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.126@tcp -r 1 -m GET -l 3 [ 794.191747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.126@tcp -r 1 -m GET -e local_error [ 794.217826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.126@tcp -r 1 -m PUT -l 6 [ 795.064397] Lustre: DEBUG MARKER: Force oleg426-client.virtnet to discover 192.168.204.126@tcp (in background) [ 795.098553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.126@tcp [ 795.900736] Lustre: DEBUG MARKER: Force oleg426-server to discover 192.168.204.26@tcp [ 797.348851] Lustre: DEBUG MARKER: Wait for 61759 [ 798.175389] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 798.998258] Lustre: DEBUG MARKER: Finished wait on 61759 [ 799.055104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 799.098480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 799.254573] LNet: 62215:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.257969] LNetError: 62215:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.267615] LNet: Removed LNI 192.168.204.26@tcp [ 799.495741] Key type .llcrypt unregistered [ 799.498228] Key type ._llcrypt unregistered [ 808.002133] Key type ._llcrypt registered [ 808.003464] Key type .llcrypt registered [ 808.769242] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:17:51 (1758734271) [ 809.039347] Key type .llcrypt unregistered [ 809.041161] Key type ._llcrypt unregistered [ 811.926210] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 812.000271] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 812.003746] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 812.007123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 814.320546] Key type ._llcrypt registered [ 814.324458] Key type .llcrypt registered [ 814.440319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.448277] alg: No test for adler32 (adler32-zlib) [ 815.296557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.349831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.366571] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 815.839226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 815.863935] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 820.555332] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:18:03 (1758734283) [ 820.644285] LNet: 64275:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.652699] LNetError: 64275:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.660915] LNet: Removed LNI 192.168.204.26@tcp [ 820.667831] LNet: Removed LNI 10.1.2.3@tcp [ 821.049481] Key type .llcrypt unregistered [ 821.051357] Key type ._llcrypt unregistered [ 824.063757] Key type ._llcrypt registered [ 824.066526] Key type .llcrypt registered [ 824.189427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 824.196349] alg: No test for adler32 (adler32-zlib) [ 825.024522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.058388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 825.078900] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 825.478816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 825.491276] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.059538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.26@tcp [ 827.103170] LNetError: Unexpected error -22 connecting to 192.168.204.26@tcp at host 192.168.204.26:988 [ 829.160434] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:18:12 (1758734292) [ 829.222820] LNet: 65393:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.227281] LNetError: 65393:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.231719] LNet: Removed LNI 192.168.204.26@tcp [ 829.240584] LNet: Removed LNI 10.1.2.3@tcp [ 829.553197] Key type .llcrypt unregistered [ 829.554932] Key type ._llcrypt unregistered [ 831.987888] Key type ._llcrypt registered [ 831.990268] Key type .llcrypt registered [ 832.114502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.126290] alg: No test for adler32 (adler32-zlib) [ 832.945626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.977170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 832.991880] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 833.311686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 833.324908] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 833.710112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.26@tcp1 --nid 192.168.204.26@tcp2 [ 835.571132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp1 192.168.204.26@tcp2 [ 835.582131] LNetError: 66223:0:(lib-move.c:2262:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.26@tcp1 to 192.168.204.26@tcp2 - Route table may be misconfigured [ 835.599737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp2 192.168.204.26@tcp1 [ 838.345600] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:18:21 (1758734301) [ 838.385077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.396986] LNet: 66791:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.400519] LNetError: 66791:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.403380] LNet: Removed LNI 192.168.204.26@tcp1 [ 839.465805] LNet: Removed LNI 192.168.204.26@tcp2 [ 839.479611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.507683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.521974] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 839.829788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 839.841025] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 840.235532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 840.263627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.26@tcp [ 840.282895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp1 192.168.204.26@tcp1 [ 840.303894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.26@tcp -d 192.168.204.26@tcp -e network_timeout [ 840.325618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -e network_timeout [ 840.348421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -e network_timeout [ 840.369717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -e network_timeout [ 840.389183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 840.400560] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 840.411276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 840.462380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 840.505163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 840.543234] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 843.277426] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:18:26 (1758734306) [ 843.316976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.328895] LNet: 67644:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.332395] LNetError: 67644:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.335332] LNetError: 67644:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 843.338380] LNet: Removed LNI 192.168.204.26@tcp [ 844.390143] LNet: Removed LNI 192.168.204.26@tcp1 [ 844.402477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.445858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 844.580140] LNet: 67687:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.797551] Key type .llcrypt unregistered [ 844.798926] Key type ._llcrypt unregistered [ 849.029949] Key type ._llcrypt registered [ 849.031333] Key type .llcrypt registered [ 849.620544] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:18:32 (1758734312) [ 849.788182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 849.798418] alg: No test for adler32 (adler32-zlib) [ 850.626514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.290238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 852.009788] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:18:35 (1758734315) [ 852.057545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.070623] LNet: 68778:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.082521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.111641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 852.125487] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 852.424110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 852.433899] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 852.803552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 852.821540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp1 [ 852.839959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp1 [ 855.550453] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:18:38 (1758734318) [ 856.343153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg426-server [ 857.078179] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:18:40 (1758734320) [ 857.891928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg426-server [ 858.704356] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:18:41 (1758734321) [ 859.495862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg426-server [ 860.258341] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:18:43 (1758734323) [ 861.066577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg426-server [ 861.788965] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:18:44 (1758734324) [ 862.591807] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg426-server [ 863.299098] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:18:46 (1758734326) [ 864.094967] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg426-server [ 864.871530] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:18:47 (1758734327) [ 865.726380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg426-server [ 866.542093] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:18:49 (1758734329) [ 867.465382] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg426-server [ 868.267898] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 13:18:51 (1758734331) [ 869.109537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg426-server [ 869.979877] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:18:53 (1758734333) [ 870.021276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.036696] LNet: 72642:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.040352] LNetError: 72642:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.043760] LNet: Removed LNI 192.168.204.26@tcp [ 871.077123] LNet: Removed LNI 192.168.204.26@tcp1 [ 871.087783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.115660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.130163] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 871.464295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 871.510939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 871.593123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.603530] LNet: 72791:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.608282] LNetError: 72791:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.612348] LNetError: 72791:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 871.620609] LNet: Removed LNI 192.168.204.26@tcp [ 871.630942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.657355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 871.670981] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 872.002945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 872.113693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.192438] LNet: Removed LNI 192.168.204.26@tcp [ 873.207511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.236051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.252433] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 873.656546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 873.703653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 873.805792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.817483] LNet: 73077:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.820737] LNet: 73077:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 873.825297] LNetError: 73077:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.827835] LNetError: 73077:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 873.837500] LNet: Removed LNI 192.168.204.26@tcp [ 873.848379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.878675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 873.893739] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 874.242619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 875.387453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.454868] LNet: Removed LNI 192.168.204.26@tcp [ 876.476700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.508764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.523200] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 876.849236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 876.888052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 876.983360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.995933] LNetError: 73373:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.999462] LNetError: 73373:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 877.009665] LNet: Removed LNI 192.168.204.26@tcp [ 877.020742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.048158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 877.069609] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 877.438919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 878.609607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.621973] LNet: 73524:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.628249] LNet: 73524:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 879.655621] LNet: Removed LNI 192.168.204.26@tcp [ 879.669148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.701094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.716849] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 880.045700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 880.085159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 880.201333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.255070] LNet: Removed LNI 192.168.204.26@tcp [ 881.266983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.293384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 881.305953] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 881.641681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 881.735954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.747951] LNetError: 73812:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.752507] LNetError: 73812:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 881.763392] LNet: Removed LNI 192.168.204.26@tcp [ 881.772915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.801820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.825060] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 882.158616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 882.195902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 883.431587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.453663] LNet: Removed LNI 192.168.204.26@tcp [ 884.467369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.500428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 884.514498] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 884.873331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 886.104430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.141404] LNet: Removed LNI 192.168.204.26@tcp [ 887.153367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.182091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 887.194254] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 887.525980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 887.568213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 887.717885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.731326] LNet: 74260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.734873] LNet: 74260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 888.805449] LNet: Removed LNI 192.168.204.26@tcp [ 888.814387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.841918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 888.855623] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 889.214718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 889.368718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.405409] LNet: Removed LNI 192.168.204.26@tcp [ 890.417051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.452269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 890.466509] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 890.819697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 890.850659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.864426] LNetError: 74532:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.868763] LNetError: 74532:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 890.879716] LNet: Removed LNI 192.168.204.26@tcp [ 890.894319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.922235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 890.938447] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 891.324258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 894.501888] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:19:17 (1758734357) [ 894.566816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.587531] LNet: Removed LNI 192.168.204.26@tcp [ 894.602454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.636908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 894.651058] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/180] [ 895.211168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.227586] LNet: Removed LNI 192.168.204.26@tcp231 [ 895.239572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.275713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 895.290534] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/180] [ 895.294514] LNet: 75376:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 895.495352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 895.509883] LNet: Removed LNI 192.168.204.26@tcp231 [ 895.517996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 895.531200] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/180] [ 896.030730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.054642] LNet: Removed LNI 192.168.204.26@tcp231 [ 896.068510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.099235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 896.112610] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/0] [ 896.115367] LNet: 75525:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 896.310719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 896.330096] LNet: Removed LNI 192.168.204.26@tcp231 [ 896.340367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 896.353482] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/0] [ 896.871190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.889283] LNet: Removed LNI 192.168.204.26@tcp231 [ 896.900457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.937362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 896.951679] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/60] [ 897.146142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 897.162564] LNet: Removed LNI 192.168.204.26@tcp231 [ 897.175141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 897.189772] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/60] [ 897.740399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.761538] LNet: Removed LNI 192.168.204.26@tcp231 [ 897.772438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.812801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 897.827962] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/180] [ 897.831540] LNet: 75821:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 897.834491] LNet: 75821:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 898.024186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 898.048371] LNet: Removed LNI 192.168.204.26@tcp231 [ 898.058326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 898.073482] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/180] [ 901.209798] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:19:24 (1758734364) [ 901.303502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.325327] LNet: Removed LNI 192.168.204.26@tcp231 [ 901.336262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.367392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 901.379524] LNet: Added LNI 192.168.204.26@tcp232 [8/256/0/180] [ 901.672212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 901.692050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 901.710447] LNet: Removed LNI 192.168.204.26@tcp232 [ 901.718259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 901.732231] LNet: Added LNI 192.168.204.26@tcp232 [8/256/0/180] [ 902.036576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 902.056732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 902.082349] LNet: Removed LNI 192.168.204.26@tcp232 [ 902.091892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 902.105432] LNet: Added LNI 192.168.204.26@tcp232 [8/256/0/180] [ 902.410816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 902.428299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 902.448482] LNet: Removed LNI 192.168.204.26@tcp232 [ 902.456973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 902.472154] LNet: Added LNI 192.168.204.26@tcp232 [8/256/0/180] [ 902.793032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 902.813773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 902.834307] LNet: Removed LNI 192.168.204.26@tcp232 [ 902.844676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 905.660217] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 13:19:28 (1758734368) [ 905.855661] LNet: 77645:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.858355] LNet: 77645:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 906.149234] Key type .llcrypt unregistered [ 906.150894] Key type ._llcrypt unregistered [ 908.530213] Key type ._llcrypt registered [ 908.533033] Key type .llcrypt registered [ 908.623283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 908.632083] alg: No test for adler32 (adler32-zlib) [ 909.458347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 909.476155] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 912.919098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 916.426595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.444321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 916.455546] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 917.127223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_interrupt -r 2 [ 917.147221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_interrupt -r 2 [ 917.167472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_interrupt -r 2 [ 917.186736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_interrupt -r 2 [ 917.210802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_interrupt -r 2 [ 917.237747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_interrupt -r 2 [ 917.259936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_interrupt -r 2 [ 917.281098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_interrupt -r 2 [ 917.300677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 917.312360] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 917.316355] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 917.471604] LNet: 78290:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 917.507071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 917.526233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 917.758543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_dropped -r 2 [ 917.777787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_dropped -r 2 [ 917.796731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_dropped -r 2 [ 917.816046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_dropped -r 2 [ 917.838963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_dropped -r 2 [ 917.860734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_dropped -r 2 [ 917.878807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_dropped -r 2 [ 917.896452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_dropped -r 2 [ 917.915766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 917.927933] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 918.094428] LNet: 78398:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 918.140336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 918.160544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 918.398076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_aborted -r 2 [ 918.417356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_aborted -r 2 [ 918.435471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_aborted -r 2 [ 918.456876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_aborted -r 2 [ 918.481409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_aborted -r 2 [ 918.504926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_aborted -r 2 [ 918.524417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_aborted -r 2 [ 918.544226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_aborted -r 2 [ 918.567220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 918.767165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 918.786875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 919.020346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_no_route -r 2 [ 919.037221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_no_route -r 2 [ 919.054293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_no_route -r 2 [ 919.073590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_no_route -r 2 [ 919.092926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_no_route -r 2 [ 919.111589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_no_route -r 2 [ 919.131540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_no_route -r 2 [ 919.152458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_no_route -r 2 [ 919.171109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 919.181776] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 919.185091] LNet: Skipped 2 previous similar messages [ 919.325873] LNet: 78615:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 919.329677] LNet: 78615:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 919.366574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 919.384906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 919.606395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_timeout -r 2 [ 919.624457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_timeout -r 2 [ 919.642708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_timeout -r 2 [ 919.662874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_timeout -r 2 [ 919.682213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_timeout -r 2 [ 919.702138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_timeout -r 2 [ 919.720337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_timeout -r 2 [ 919.739427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_timeout -r 2 [ 919.758438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 919.942514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 919.959955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 920.196052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_dropped -r 2 [ 920.215538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e remote_dropped -r 2 [ 920.237607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_dropped -r 2 [ 920.256939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e remote_dropped -r 2 [ 920.279559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e remote_dropped -r 2 [ 920.300232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e remote_dropped -r 2 [ 920.319964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e remote_dropped -r 2 [ 920.341600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e remote_dropped -r 2 [ 920.364393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 920.375137] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.126@tcp [ 920.547312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 920.565551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 922.907092] LNet: 78899:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.910444] LNetError: 78899:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.913840] LNet: Removed LNI 192.168.204.26@tcp [ 922.920469] LNet: Removed LNI 192.168.204.26@tcp1 [ 923.111116] Key type .llcrypt unregistered [ 923.112905] Key type ._llcrypt unregistered [ 929.457653] Key type ._llcrypt registered [ 929.459303] Key type .llcrypt registered [ 930.040366] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:19:53 (1758734393) [ 930.193639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 930.201710] alg: No test for adler32 (adler32-zlib) [ 931.037741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.064331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 931.082177] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 931.404193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 931.440307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 931.465824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 931.498334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 931.520907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 931.539371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 934.368987] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:19:57 (1758734397) [ 934.441525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.455633] LNet: 80450:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.459870] LNetError: 80450:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.470711] LNet: Removed LNI 192.168.204.26@tcp [ 934.481604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.511238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 934.528431] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 937.766124] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:20:00 (1758734400) [ 938.469395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 939.252825] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:20:02 (1758734402) [ 939.488927] LNet: 81482:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.492705] LNetError: 81482:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.502711] LNet: Removed LNI 192.168.204.26@tcp [ 939.821817] Key type .llcrypt unregistered [ 939.823402] Key type ._llcrypt unregistered [ 942.415325] Key type ._llcrypt registered [ 942.417333] Key type .llcrypt registered [ 942.510638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.520286] alg: No test for adler32 (adler32-zlib) [ 943.353234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 943.371863] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 947.125973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 949.195837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.126@tcp [ 950.386039] LNet: 81893:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.391553] LNetError: 81893:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.404578] LNet: Removed LNI 192.168.204.26@tcp [ 950.596429] Key type .llcrypt unregistered [ 950.597961] Key type ._llcrypt unregistered [ 955.118640] Key type ._llcrypt registered [ 955.120123] Key type .llcrypt registered [ 955.680725] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 956.321933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 957.011076] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:20:20 (1758734420) [ 957.146923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 957.154070] alg: No test for adler32 (adler32-zlib) [ 957.969423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.008950] LNet: 82975:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.283473] Key type .llcrypt unregistered [ 958.285129] Key type ._llcrypt unregistered [ 960.676527] Key type ._llcrypt registered [ 960.677940] Key type .llcrypt registered [ 960.753063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.763321] alg: No test for adler32 (adler32-zlib) [ 961.590219] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 961.592371] LNet: 83227:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 961.609652] LNetError: 83239:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.28@tcp is being used as a gateway but routing feature is not turned on [ 961.622981] LNetError: 83239:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.204.28@tcp has gone from up to down [ 961.827721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.895153] LNetError: 83239:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.204.27@tcp has gone from up to down [ 964.899715] LNetError: 83239:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 964.904159] LNet: 83285:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.907803] LNetError: 83285:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.922525] LNet: Removed LNI 192.168.204.26@tcp [ 964.937841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.968444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 964.981441] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 964.985444] LNet: 83293:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 964.992396] LNet: 83293:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 964.996535] LNetError: 83293:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.28@tcp is a gateway. Can not delete it [ 965.001504] LNetError: 83293:0:(peer.c:2201:lnet_del_peer_ni()) 192.168.204.28@tcp is a router. Can not be deleted [ 965.009723] LNetError: 83290:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.30@tcp is being used as a gateway but routing feature is not turned on [ 965.015549] LNetError: 83290:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 965.020489] LNet: 83293:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 967.772506] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:20:30 (1758734430) [ 967.776362] LNetError: 83295:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.204.27@tcp has gone from up to down [ 967.788442] LNetError: 83295:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 968.585507] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg426-server [ 969.363331] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 13:20:32 (1758734432) [ 970.155209] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg426-server [ 970.876393] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:20:34 (1758734434) [ 971.564601] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 973.256530] Key type lgssc registered [ 973.806790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 983.404580] Key type lgssc unregistered [ 983.598629] LNet: 85711:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.603223] LNetError: 85711:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.615353] LNet: Removed LNI 192.168.204.26@tcp [ 983.912343] Key type .llcrypt unregistered [ 983.914156] Key type ._llcrypt unregistered [ 990.167929] Key type ._llcrypt registered [ 990.169202] Key type .llcrypt registered [ 990.738878] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 13:20:53 (1758734453) [ 990.896959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 990.903433] alg: No test for adler32 (adler32-zlib) [ 991.731021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.769931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 991.795761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 991.824119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 991.848170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 994.675356] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 13:20:57 (1758734457) [ 994.735730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.750076] LNet: 87110:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.762286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.792035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 994.805595] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 995.227874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.239446] LNetError: 87254:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.256541] LNet: Removed LNI 192.168.204.26@tcp [ 995.267972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.295555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 995.308276] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 995.524770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.534734] LNet: 87282:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.537641] LNet: 87282:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 995.545719] LNet: Removed LNI 192.168.204.26@tcp [ 995.555688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.587664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 995.600811] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 998.635141] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 13:21:01 (1758734461) [ 998.688281] LNet: 87868:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.691744] LNetError: 87868:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.695227] LNetError: 87868:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 998.704729] LNet: Removed LNI 192.168.204.26@tcp [ 998.996822] Key type .llcrypt unregistered [ 998.998550] Key type ._llcrypt unregistered [ 1001.513790] Key type ._llcrypt registered [ 1001.515309] Key type .llcrypt registered [ 1001.597666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.606558] alg: No test for adler32 (adler32-zlib) [ 1002.421068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.456290] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1002.458525] LNet: *** cfs_fail_loc=e003, val=100*** [ 1002.765647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1002.775587] Ping reply received with mapping: [ 1002.777123] found metadata reply NI[192.168.204.26@tcp] [ 1002.778458] 1a cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1002.780629] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1002.781887] 5a 5a 5a 5a 5a 5a 5a 5a [ 1002.833275] LNet: 88262:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.836340] LNetError: 88262:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.848544] LNet: Removed LNI 192.168.204.26@tcp [ 1003.124976] Key type .llcrypt unregistered [ 1003.126374] Key type ._llcrypt unregistered [ 1007.349303] Key type ._llcrypt registered [ 1007.351033] Key type .llcrypt registered [ 1007.891428] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:21:11 (1758734471) [ 1008.093717] Key type .llcrypt unregistered [ 1008.095341] Key type ._llcrypt unregistered [ 1010.405603] Key type ._llcrypt registered [ 1010.407230] Key type .llcrypt registered [ 1010.496840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1010.504200] alg: No test for adler32 (adler32-zlib) [ 1011.423159] LNetError: 89296:0:(api-ni.c:2709:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1011.594551] Key type .llcrypt unregistered [ 1011.596095] Key type ._llcrypt unregistered [ 1013.774587] Key type ._llcrypt registered [ 1013.776168] Key type .llcrypt registered [ 1014.392540] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 13:21:17 (1758734477) [ 1014.612541] Key type .llcrypt unregistered [ 1014.614412] Key type ._llcrypt unregistered [ 1016.905107] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1016.907255] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1019.143829] Key type ._llcrypt registered [ 1019.145309] Key type .llcrypt registered [ 1019.238993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1019.247903] alg: No test for adler32 (adler32-zlib) [ 1020.075872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.106376] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.696960] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 13:21:27 (1758734487) [ 1024.733850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.747140] LNet: 91056:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.751035] LNetError: 91056:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.759842] LNet: Removed LNI 10.1.2.3@tcp [ 1024.771712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.113525] Key type .llcrypt unregistered [ 1025.115308] Key type ._llcrypt unregistered [ 1027.465941] Key type ._llcrypt registered [ 1027.467482] Key type .llcrypt registered [ 1027.542389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.550092] alg: No test for adler32 (adler32-zlib) [ 1028.379618] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1028.411586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1028.489691] LNet: 91347:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.493605] LNetError: 91347:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.500502] LNet: Removed LNI 192.168.204.26@tcp [ 1028.793626] Key type .llcrypt unregistered [ 1028.795442] Key type ._llcrypt unregistered [ 1033.095231] Key type ._llcrypt registered [ 1033.096720] Key type .llcrypt registered [ 1033.666393] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:21:36 (1758734496) [ 1033.903514] Key type .llcrypt unregistered [ 1033.905038] Key type ._llcrypt unregistered [ 1036.205661] Key type ._llcrypt registered [ 1036.207201] Key type .llcrypt registered [ 1036.287184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1036.293758] alg: No test for adler32 (adler32-zlib) [ 1039.930483] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:21:43 (1758734503) [ 1039.967108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1039.986099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.023921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1040.035880] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1040.357746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1040.378060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1040.393621] LNet: Removed LNI 192.168.204.26@tcp [ 1040.506840] LNet: 93077:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.688439] Key type .llcrypt unregistered [ 1040.690388] Key type ._llcrypt unregistered [ 1045.538373] Key type ._llcrypt registered [ 1045.539667] Key type .llcrypt registered [ 1046.124532] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:21:49 (1758734509) [ 1046.282263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1046.291592] alg: No test for adler32 (adler32-zlib) [ 1047.107710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.135508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1047.147923] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1047.471483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1047.489146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.26@tcp [ 1047.498362] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1047.500293] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1047.502070] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1047, 0, 0 [ 1047.504833] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1047.506328] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1047.508083] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1047.509872] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1047, 0, 0 [ 1047.512701] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1047.514571] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1047.518114] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1047.520202] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1047, 0, 1097 [ 1047.523764] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1047.524947] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1047.526994] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1047.528892] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1047, 0, 1097 [ 1047.531840] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1050.295574] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:21:53 (1758734513) [ 1050.492879] LNet: 94610:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.496813] LNetError: 94610:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.505522] LNet: Removed LNI 192.168.204.26@tcp [ 1050.795462] Key type .llcrypt unregistered [ 1050.797089] Key type ._llcrypt unregistered [ 1053.190535] Key type ._llcrypt registered [ 1053.192145] Key type .llcrypt registered [ 1053.275438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1053.281968] alg: No test for adler32 (adler32-zlib) [ 1054.106965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1054.122138] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1057.466832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1060.779093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.796112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1060.807469] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1061.209384] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1061.245299] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1061.248110] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1061.251996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1063.426684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1063.436566] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1068.225847] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:22:11 (1758734531) [ 1068.314275] LNet: 96101:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.317497] LNetError: 96101:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.321929] LNet: Removed LNI 192.168.204.26@tcp [ 1069.343251] LNet: Removed LNI 192.168.204.26@tcp1 [ 1069.345114] LNetError: 96101:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1069.348253] LNetError: 96101:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1069.357962] LNet: Removed LNI 10.1.2.3@tcp99 [ 1069.675622] Key type .llcrypt unregistered [ 1069.677861] Key type ._llcrypt unregistered [ 1072.128119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1072.130188] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1074.342041] Key type ._llcrypt registered [ 1074.343446] Key type .llcrypt registered [ 1074.424046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1074.431420] alg: No test for adler32 (adler32-zlib) [ 1075.247857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1075.281909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1075.300196] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1075.626153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1075.636288] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1076.641987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1076.666064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1076.688875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1076.699073] LNetError: 96835:0:(peer.c:2209:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1076.713979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1076.736881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 1076.931081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1076.968358] LNet: 96855:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.971709] LNetError: 96855:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.974559] LNet: Removed LNI 192.168.204.26@tcp [ 1076.985449] LNet: Removed LNI 10.1.2.3@tcp [ 1077.278617] Key type .llcrypt unregistered [ 1077.280192] Key type ._llcrypt unregistered [ 1081.863835] Key type ._llcrypt registered [ 1081.865414] Key type .llcrypt registered [ 1082.435753] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:22:25 (1758734545) [ 1082.594383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1082.602095] alg: No test for adler32 (adler32-zlib) [ 1083.423892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.450751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1083.464630] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1083.781113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1083.819563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg426-client@tcp [ 1086.611341] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:22:29 (1758734549) [ 1086.664960] LNet: 98375:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.670395] LNetError: 98375:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.678771] LNet: Removed LNI 192.168.204.26@tcp [ 1086.975427] Key type .llcrypt unregistered [ 1086.977109] Key type ._llcrypt unregistered [ 1089.407296] Key type ._llcrypt registered [ 1089.408878] Key type .llcrypt registered [ 1089.482627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1089.493319] alg: No test for adler32 (adler32-zlib) [ 1090.325375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1090.346582] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1090.366850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1093.317059] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:22:36 (1758734556) [ 1094.007651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1094.743595] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:22:37 (1758734557) [ 1095.451416] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1096.285268] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:22:39 (1758734559) [ 1096.336223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1096.348491] LNet: 99789:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.351369] LNetError: 99789:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.358357] LNet: Removed LNI 192.168.204.26@tcp [ 1096.370537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1096.400031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1096.413853] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1096.762199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1096.789465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1099.934296] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:22:42 (1758734562) [ 1099.970383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1099.986464] LNet: 100492:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.990043] LNetError: 100492:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.998366] LNet: Removed LNI 192.168.204.26@tcp [ 1100.014602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.046315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1100.061384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1100.383469] Key type .llcrypt unregistered [ 1100.385335] Key type ._llcrypt unregistered [ 1105.225250] Key type ._llcrypt registered [ 1105.226950] Key type .llcrypt registered [ 1105.852434] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:22:48 (1758734568) [ 1106.033408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1106.040778] alg: No test for adler32 (adler32-zlib) [ 1106.863292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.889216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1106.903556] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1107.261569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1107.292496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 1110.347168] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:22:53 (1758734573) [ 1110.443792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1110.459945] LNet: 102045:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.466144] LNetError: 102045:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.475548] LNet: Removed LNI 192.168.204.26@tcp [ 1110.489268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1110.519982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1110.537552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1113.676564] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1114.462095] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:22:57 (1758734577) [ 1114.564361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1114.574557] LNet: 102765:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.587315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1114.703622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1114.714307] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1115.044060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1115.055514] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1115.423263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 1115.441315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1115.468987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.26@tcp -m PUT -e local_timeout -r 1 [ 1115.498284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1115.525119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1115.527282] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1115.628580] LNetError: 102797:0:(lib-move.c:4945:lnet_parse()) 192.168.204.26@tcp, src 192.168.204.26@tcp: Dropping PUT (error -108 looking up sender) [ 1116.639691] LNet: 103072:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.643480] LNetError: 103072:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1117.663116] LNet: Removed LNI 192.168.204.26@tcp [ 1117.664899] LNetError: 103072:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1117.675468] LNet: Removed LNI 10.1.2.3@tcp [ 1117.877841] Key type .llcrypt unregistered [ 1117.879668] Key type ._llcrypt unregistered [ 1122.678738] Key type ._llcrypt registered [ 1122.680308] Key type .llcrypt registered [ 1123.318995] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:23:06 (1758734586) [ 1123.503254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1123.510418] alg: No test for adler32 (adler32-zlib) [ 1124.329778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1124.571500] LNet: 103920:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.909376] Key type .llcrypt unregistered [ 1124.911073] Key type ._llcrypt unregistered [ 1127.470680] Key type ._llcrypt registered [ 1127.472925] Key type .llcrypt registered [ 1127.585689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1127.594030] alg: No test for adler32 (adler32-zlib) [ 1128.451636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1128.477213] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1132.350192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1132.448455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1132.772340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.126@tcp [ 1132.947429] LNet: 104356:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1132.988644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.006973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.311367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1133.524564] LNet: 104461:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.565322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.584036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.912593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.126@tcp [ 1134.112542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1134.131624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.453025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1134.623531] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.126@tcp [ 1134.623604] LNet: 104672:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1134.630915] LNet: 104672:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 1134.671382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1134.691504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.885747] LNet: 104713:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.888965] LNetError: 104713:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.900404] LNet: Removed LNI 192.168.204.26@tcp [ 1135.101536] Key type .llcrypt unregistered [ 1135.103309] Key type ._llcrypt unregistered [ 1142.405055] Key type ._llcrypt registered [ 1142.406863] Key type .llcrypt registered [ 1143.063456] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:23:26 (1758734606) [ 1143.286633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1143.294760] alg: No test for adler32 (adler32-zlib) [ 1144.133882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1144.325575] LNet: 105595:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1144.660465] Key type .llcrypt unregistered [ 1144.662103] Key type ._llcrypt unregistered [ 1147.189620] Key type ._llcrypt registered [ 1147.190865] Key type .llcrypt registered [ 1147.290319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1147.299464] alg: No test for adler32 (adler32-zlib) [ 1148.131846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1148.147669] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1151.693814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1155.064412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.097403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1155.109274] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1155.548154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1155.798666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.126@tcp [ 1155.940170] LNet: 106230:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.974931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.991081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.279971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1156.383459] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.126@tcp, 192.168.204.126@tcp1 [ 1156.470848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.488522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.789958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.126@tcp [ 1156.944206] LNet: 106440:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.948157] LNet: 106440:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 1156.988205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.006127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.279291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1157.464036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.485406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.964622] LNet: 106620:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.968389] LNetError: 106620:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.972154] LNet: Removed LNI 192.168.204.26@tcp [ 1159.977431] LNet: Removed LNI 192.168.204.26@tcp1 [ 1160.163741] Key type .llcrypt unregistered [ 1160.165506] Key type ._llcrypt unregistered [ 1166.525342] Key type ._llcrypt registered [ 1166.526828] Key type .llcrypt registered [ 1167.110988] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 961 sec =========== 13:23:50 (1758734630) [ 1167.417775] Key type .llcrypt unregistered [ 1167.419519] Key type ._llcrypt unregistered [ 1172.410518] Key type ._llcrypt registered [ 1172.411804] Key type .llcrypt registered [ 1172.602838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1172.610290] alg: No test for adler32 (adler32-zlib) [ 1173.505343] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 1173.626177] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1175.233606] Key type lgssc registered [ 1175.775987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1204.231335] Lustre: Mounted lustre-client [ 1207.094260] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1215.464435] LustreError: 109833:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8f0941370800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1215.478279] LustreError: 109833:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1215.501426] Lustre: Unmounted lustre-client [ 1229.303451] Key type lgssc unregistered [ 1229.450863] LNet: 110314:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.453312] LNetError: 110314:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.462831] LNet: Removed LNI 192.168.204.26@tcp [ 1229.772210] Key type .llcrypt unregistered [ 1229.773948] Key type ._llcrypt unregistered