[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 389445955 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002118] x2apic enabled [ 0.002865] Switched APIC routing to physical x2apic. [ 0.003009] kvm-guest: setup PV IPIs [ 0.005216] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007008] pid_max: default: 32768 minimum: 301 [ 0.008099] LSM: Security Framework initializing [ 0.009022] Yama: becoming mindful. [ 0.010017] SELinux: Initializing. [ 0.010679] *** VALIDATE selinux *** [ 0.017454] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021680] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025085] *** VALIDATE tmpfs *** [ 0.026198] *** VALIDATE proc *** [ 0.027173] *** VALIDATE cgroup *** [ 0.027873] *** VALIDATE cgroup2 *** [ 0.028235] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032004] Spectre V2 : User space: Vulnerable [ 0.033006] Speculative Store Bypass: Vulnerable [ 0.035769] debug: unmapping init [mem 0xffffffff8d459000-0xffffffff8d460fff] [ 0.037157] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038667] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039023] ... version: 2 [ 0.040010] ... bit width: 48 [ 0.041010] ... generic registers: 4 [ 0.042009] ... value mask: 0000ffffffffffff [ 0.043007] ... max period: 00007fffffffffff [ 0.043961] ... fixed-purpose events: 3 [ 0.044008] ... event mask: 000000070000000f [ 0.045170] rcu: Hierarchical SRCU implementation. [ 0.047143] smp: Bringing up secondary CPUs ... [ 0.048388] x86: Booting SMP configuration: [ 0.049013] .... node #0, CPUs: #1 #2 #3 [ 0.051383] smp: Brought up 1 node, 4 CPUs [ 0.052759] smpboot: Max logical packages: 1 [ 0.053011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.278519] node 0 deferred pages initialised in 224ms [ 0.281117] devtmpfs: initialized [ 0.282201] x86/mm: Memory block size: 128MB [ 0.284774] gcov: version magic: 0x41383552 [ 0.285547] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.286046] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.287158] pinctrl core: initialized pinctrl subsystem [ 0.288130] [ 0.288688] ************************************************************* [ 0.289008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290006] ** ** [ 0.291005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.292007] ** ** [ 0.293005] ** This means that this kernel is built to expose internal ** [ 0.294005] ** IOMMU data structures, which may compromise security on ** [ 0.295005] ** your system. ** [ 0.296005] ** ** [ 0.297006] ** If you see this message and you are not debugging the ** [ 0.298006] ** kernel, report this immediately to your vendor! ** [ 0.299008] ** ** [ 0.300008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.301006] ************************************************************* [ 0.302571] NET: Registered protocol family 16 [ 0.303394] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.304032] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.305032] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.306357] cpuidle: using governor menu [ 0.307000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.310469] PCI: Using configuration type 1 for base access [ 0.312113] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.319072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.321020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.324033] cryptd: max_cpu_qlen set to 1000 [ 0.326145] ACPI: Added _OSI(Module Device) [ 0.328010] ACPI: Added _OSI(Processor Device) [ 0.328735] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.330013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.334257] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.338556] ACPI: Interpreter enabled [ 0.340061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.342009] ACPI: Using IOAPIC for interrupt routing [ 0.343088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.346369] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.357683] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.359020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.362010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.365055] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.370256] acpiphp: Slot [2] registered [ 0.371119] acpiphp: Slot [3] registered [ 0.372111] acpiphp: Slot [4] registered [ 0.374078] acpiphp: Slot [5] registered [ 0.375091] acpiphp: Slot [6] registered [ 0.376076] acpiphp: Slot [7] registered [ 0.377055] acpiphp: Slot [8] registered [ 0.378055] acpiphp: Slot [9] registered [ 0.380055] acpiphp: Slot [10] registered [ 0.381106] acpiphp: Slot [11] registered [ 0.382071] acpiphp: Slot [12] registered [ 0.383059] acpiphp: Slot [13] registered [ 0.384059] acpiphp: Slot [14] registered [ 0.385057] acpiphp: Slot [15] registered [ 0.386059] acpiphp: Slot [16] registered [ 0.387086] acpiphp: Slot [17] registered [ 0.388057] acpiphp: Slot [18] registered [ 0.389056] acpiphp: Slot [19] registered [ 0.390056] acpiphp: Slot [20] registered [ 0.391054] acpiphp: Slot [21] registered [ 0.392041] acpiphp: Slot [22] registered [ 0.393055] acpiphp: Slot [23] registered [ 0.394076] acpiphp: Slot [24] registered [ 0.396064] acpiphp: Slot [25] registered [ 0.397073] acpiphp: Slot [26] registered [ 0.399093] acpiphp: Slot [27] registered [ 0.400063] acpiphp: Slot [28] registered [ 0.401078] acpiphp: Slot [29] registered [ 0.403127] acpiphp: Slot [30] registered [ 0.404091] acpiphp: Slot [31] registered [ 0.406092] PCI host bridge to bus 0000:00 [ 0.408020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.410017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.412020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.415018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.417010] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.418012] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.420185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.422976] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.426788] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.431408] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.434495] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.437012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.439009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.441009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.442545] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.445683] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.447024] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.449606] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.452012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.458000] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.463015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.468768] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.474027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.479019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.497020] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.504320] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.507980] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.514012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.524011] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.531000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.533220] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.535812] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.538296] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.540195] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.545135] iommu: Default domain type: Passthrough [ 0.546380] SCSI subsystem initialized [ 0.548085] ACPI: bus type USB registered [ 0.549086] usbcore: registered new interface driver usbfs [ 0.550050] usbcore: registered new interface driver hub [ 0.552056] usbcore: registered new device driver usb [ 0.553156] pps_core: LinuxPPS API ver. 1 registered [ 0.555010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.557079] PTP clock support registered [ 0.561128] EDAC MC: Ver: 3.0.0 [ 0.562412] PCI: Using ACPI for IRQ routing [ 0.564685] NetLabel: Initializing [ 0.566007] NetLabel: domain hash size = 128 [ 0.567007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.569074] NetLabel: unlabeled traffic allowed by default [ 0.570122] vgaarb: loaded [ 0.571213] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.573009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.576594] clocksource: Switched to clocksource kvm-clock [ 0.670299] VFS: Disk quotas dquot_6.6.0 [ 0.671578] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.673581] *** VALIDATE ramfs *** [ 0.674573] *** VALIDATE hugetlbfs *** [ 0.675833] pnp: PnP ACPI init [ 0.678030] pnp: PnP ACPI: found 6 devices [ 0.692679] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.694845] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.696070] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.697088] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.698245] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.699494] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.701056] NET: Registered protocol family 2 [ 0.702689] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.706562] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.708351] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.711704] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.713989] TCP: Hash tables configured (established 65536 bind 65536) [ 0.716109] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.718251] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.720270] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.722333] NET: Registered protocol family 1 [ 0.724284] RPC: Registered named UNIX socket transport module. [ 0.725509] RPC: Registered udp transport module. [ 0.726377] RPC: Registered tcp transport module. [ 0.727295] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.728401] NET: Registered protocol family 44 [ 0.729252] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.730374] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.731403] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.732528] PCI: CLS 0 bytes, default 64 [ 0.733425] Unpacking initramfs... [ 2.082900] debug: unmapping init [mem 0xffffa0e0fcc64000-0xffffa0e0fffcffff] [ 2.088901] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.091276] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.094840] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.675785] Initialise system trusted keyrings [ 2.677516] Key type blacklist registered [ 2.679458] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.688520] zbud: loaded [ 2.692345] *** VALIDATE nfs *** [ 2.693634] *** VALIDATE nfs4 *** [ 2.695371] pstore: using deflate compression [ 2.698879] Platform Keyring initialized [ 2.819140] NET: Registered protocol family 38 [ 2.822320] Key type asymmetric registered [ 2.825122] Asymmetric key parser 'x509' registered [ 2.827715] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.831359] io scheduler mq-deadline registered [ 2.833723] io scheduler kyber registered [ 2.836095] io scheduler bfq registered [ 2.838254] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.842623] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.846939] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.851227] ACPI: Power Button [PWRF] [ 2.964844] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.066812] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.187833] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.222689] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.258174] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.273740] Non-volatile memory driver v1.3 [ 3.274977] Linux agpgart interface v0.103 [ 3.320308] virtio_blk virtio1: [vda] 133152 512-byte logical blocks (68.2 MB/65.0 MiB) [ 3.325262] vda: detected capacity change from 0 to 68173824 [ 3.345985] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.350046] vdb: detected capacity change from 0 to 1073741824 [ 3.357874] libphy: Fixed MDIO Bus: probed [ 3.368816] usbcore: registered new interface driver usbserial_generic [ 3.370956] usbserial: USB Serial support registered for generic [ 3.374409] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.383760] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.385636] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.388168] mousedev: PS/2 mouse device common for all mice [ 3.392070] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.396372] rtc_cmos 00:05: RTC can wake from S4 [ 3.400576] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.403681] rtc_cmos 00:05: registered as rtc0 [ 3.405165] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.411448] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.415598] intel_pstate: CPU model not supported [ 3.418381] hid: raw HID events driver (C) Jiri Kosina [ 3.420687] usbcore: registered new interface driver usbhid [ 3.422373] usbhid: USB HID core driver [ 3.423686] drop_monitor: Initializing network drop monitor service [ 3.425337] Initializing XFRM netlink socket [ 3.426843] NET: Registered protocol family 10 [ 3.429110] Segment Routing with IPv6 [ 3.430380] NET: Registered protocol family 17 [ 3.433140] mpls_gso: MPLS GSO support [ 3.442235] RAS: Correctable Errors collector initialized. [ 3.444159] AVX version of gcm_enc/dec engaged. [ 3.445913] AES CTR mode by8 optimization enabled [ 3.580869] sched_clock: Marking stable (3580810056, 0)->(4372097443, -791287387) [ 3.585771] registered taskstats version 1 [ 3.592677] Loading compiled-in X.509 certificates [ 3.594232] zswap: loaded using pool lzo/zbud [ 3.639189] Key type big_key registered [ 3.655328] Key type encrypted registered [ 3.657303] ima: No TPM chip found, activating TPM-bypass! [ 3.658686] ima: Allocated hash algorithm: sha1 [ 3.659785] ima: No architecture policies found [ 3.661314] evm: Initialising EVM extended attributes: [ 3.662659] evm: security.selinux [ 3.663582] evm: security.ima [ 3.664421] evm: security.capability [ 3.665487] evm: HMAC attrs: 0x1 [ 3.672176] rtc_cmos 00:05: setting system clock to 2025-09-25 13:43:12 UTC (1758807792) [ 3.680082] debug: unmapping init [mem 0xffffffff8e403000-0xffffffff8e5fffff] [ 3.683979] debug: unmapping init [mem 0xffffffff8d182000-0xffffffff8d458fff] [ 3.697264] Write protecting the kernel read-only data: 28672k [ 3.709566] debug: unmapping init [mem 0xffffffff8b803000-0xffffffff8b9fffff] [ 3.715331] debug: unmapping init [mem 0xffffffff8c114000-0xffffffff8c1fffff] [ 3.770419] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.780414] systemd[1]: Detected virtualization kvm. [ 3.782567] systemd[1]: Detected architecture x86-64. [ 3.784496] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.822173] systemd[1]: No hostname configured. [ 3.824840] systemd[1]: Set hostname to . [ 3.827939] random: systemd: uninitialized urandom read (16 bytes read) [ 3.835807] systemd[1]: Initializing machine ID from random generator. [ 3.957498] random: ln: uninitialized urandom read (6 bytes read) [ 4.094782] random: systemd: uninitialized urandom read (16 bytes read) [ 4.099591] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.109072] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.125708] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.076757] device-mapper: uevent: version 1.0.3 [ 5.081438] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.554063] random: fast init done [ 6.648735] virtio_net virtio0 ens2: renamed from eth0 [ 7.101177] scsi host0: ata_piix [ 7.115366] scsi host1: ata_piix [ 7.116586] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.132364] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.247967] random: crng init done [ 12.249803] random: 7 urandom warning(s) missed due to ratelimiting [ 12.442632] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.325431] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.042475] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.470165] SELinux: Disabled at runtime. [ 17.543911] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.556632] systemd[1]: Detected virtualization kvm. [ 17.558915] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.586873] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.591312] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.603927] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.613637] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.622558] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.651858] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.678557] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables...[ 18.928197] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 20.372746] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.442337] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.463103] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.334769] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.428743] EDAC sbridge: Ver: 1.1.2 [ 24.804783] Key type dns_resolver registered [ 25.453508] NFS: Registering the id_resolver key type [ 25.454766] Key type id_resolver registered [ 25.455671] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg223-client login: [ 71.530502] libcfs: loading out-of-tree module taints kernel. [ 71.578283] Key type ._llcrypt registered [ 71.580567] Key type .llcrypt registered [ 71.863778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 71.870836] alg: No test for adler32 (adler32-zlib) [ 72.859470] Lustre: Lustre: Build Version: 2.16.58_52_ga6ec4c6 [ 73.129860] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 74.729178] Key type lgssc registered [ 75.333275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.533176] Lustre: Mounted lustre-client [ 134.886869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 146.764428] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing check_logdir /tmp/testlogs/ [ 148.539623] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing yml_node [ 150.201299] Lustre: DEBUG MARKER: Client: 2.16.58.52 [ 151.039489] Lustre: DEBUG MARKER: MDS: 2.16.58.52 [ 151.887555] Lustre: DEBUG MARKER: OSS: 2.16.58.52 [ 152.437974] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 25 09:45:40 EDT 2025 [ 157.957053] LustreError: 7911:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffffa0e150579800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 157.962140] LustreError: 7911:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 157.975098] Lustre: Unmounted lustre-client [ 171.010417] Key type lgssc unregistered [ 171.153792] LNet: 8357:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 171.157485] LNetError: 8357:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 171.168722] LNet: Removed LNI 192.168.202.23@tcp [ 171.485251] Key type .llcrypt unregistered [ 171.487085] Key type ._llcrypt unregistered [ 181.614343] Key type ._llcrypt registered [ 181.615878] Key type .llcrypt registered [ 181.702768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 181.709463] alg: No test for adler32 (adler32-zlib) [ 182.521091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 182.552184] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 182.563670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 184.267979] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 184.330907] LNet: 9223:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 184.334674] LNetError: 9223:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 184.345527] LNet: Removed LNI 192.168.202.23@tcp [ 184.606342] Key type .llcrypt unregistered [ 184.607842] Key type ._llcrypt unregistered [ 187.005618] Key type ._llcrypt registered [ 187.007191] Key type .llcrypt registered [ 187.636838] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:46:15 (1758807975) [ 187.798120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 187.806859] alg: No test for adler32 (adler32-zlib) [ 188.628205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 188.674142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 188.691034] LNet: 9634:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 188.703541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 188.736475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 188.748423] LNet: 9641:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 191.699671] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:46:19 (1758807979) [ 191.757633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.770801] LNet: 10217:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 191.781487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.818209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 194.524106] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:46:22 (1758807982) [ 194.577340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 194.589838] LNet: 10798:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 194.602775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 194.642945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 197.414691] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:46:25 (1758807985) [ 197.467822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 197.477978] LNet: 11381:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 197.488957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.525288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 200.273652] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:46:28 (1758807988) [ 200.327338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.350316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.385126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 200.565769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 203.249037] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:46:31 (1758807991) [ 203.314603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 203.324867] LNet: 12556:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 203.328063] LNet: 12556:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 203.340460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 203.381805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 203.566292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 203.746646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 206.335775] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:46:34 (1758807994) [ 206.421718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.441131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.485287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 209.224939] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:46:37 (1758807997) [ 209.258668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.278887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.311787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 209.328627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 209.502670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 209.519408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 209.687666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 209.711861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 209.877967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 209.896605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 210.070588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 210.093142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 210.262607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 210.278797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 210.460727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 210.476890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 210.642467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 210.658273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 210.837059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 210.851674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 211.012893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 211.030332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 211.203865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 211.219552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 211.382557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 211.400740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 211.577385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 211.601806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 214.393169] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:46:42 (1758808002) [ 214.447672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.458839] LNet: 14416:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 214.461813] LNet: 14416:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 214.474966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.518109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 214.537835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 217.368550] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:46:45 (1758808005) [ 217.422219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.443909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.479419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 217.496881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 220.474959] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:46:48 (1758808008) [ 220.521218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.542540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.583558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 220.598118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 223.393754] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:46:51 (1758808011) [ 223.467904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.490190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.532460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 223.549842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 226.407616] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:46:54 (1758808014) [ 226.466306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.487367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.525158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 226.542787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 229.273543] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:46:57 (1758808017) [ 229.327063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.346262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.382550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 229.399862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 232.000494] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:47:00 (1758808020) [ 232.035402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.045449] LNet: 17925:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 232.047997] LNet: 17925:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 232.060657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.112186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 232.294301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 235.056483] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:47:03 (1758808023) [ 235.104664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.124101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.212116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.314977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 238.078605] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:47:06 (1758808026) [ 238.121440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.140027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.199980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 238.304971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 241.116880] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:47:09 (1758808029) [ 241.163263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.185178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.380058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 241.488887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 244.178795] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:47:12 (1758808032) [ 244.233260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.253695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.286725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 244.310464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 246.923684] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:47:15 (1758808035) [ 246.983769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.002426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.036249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 247.057732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 249.831516] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:47:18 (1758808038) [ 249.869716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.889198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.926173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 252.347606] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:47:20 (1758808040) [ 252.377917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.395627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.431029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 255.059897] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:47:23 (1758808043) [ 255.098028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.123052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.163497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 255.346041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 258.036752] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:47:26 (1758808046) [ 258.092648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.114871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.153559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 258.170728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 260.629344] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:47:28 (1758808048) [ 260.662034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.683197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.722830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 260.738463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 263.286387] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:47:31 (1758808051) [ 263.343267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 263.365361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 263.398604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 263.416673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 266.111664] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:47:34 (1758808054) [ 266.157723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.168838] LNet: 25118:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 266.172091] LNet: 25118:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 266.182784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.222191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 266.241056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 268.924661] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:47:37 (1758808057) [ 268.959944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.981447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.015146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 269.042606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 269.052461] LNetError: 25721:0:(peer.c:2209:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 269.071925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 269.088684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 271.567287] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:47:39 (1758808059) [ 271.626534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.645984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.684209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.cm1dUmWzT9 [ 271.718733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.cm1dUmWzT9 [ 271.745102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.cm1dUmWzT9 [ 274.213248] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:47:42 (1758808062) [ 274.272293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.293911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.321836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 274.338698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 274.359665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 276.793207] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:47:45 (1758808065) [ 276.836851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.858845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.893881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 278.024325] LNet: Message forwarding will be enabled in 2s [ 281.120220] LNet: Message forwarding enabled [ 281.201771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 281.212875] LNet: Message forwarding will stop in 2s to 6s [ 284.372850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 284.958438] LNet: Message forwarding disabled [ 285.471906] LNet: Message forwarding will be enabled in 2s [ 287.938957] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:47:56 (1758808076) [ 287.995779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.622762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.662950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 291.047401] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:47:59 (1758808079) [ 291.082148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 291.101136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.131997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 291.147252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 291.162737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 291.177789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 291.193362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 291.210390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 291.225819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 291.242266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 291.260441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 291.276898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 291.293346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 291.308518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 291.325643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 291.341752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 291.359937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 291.376216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 291.393699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 291.410828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 291.425996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 291.436884] LNetError: 28777:0:(router.c:628:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 293.718980] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:48:02 (1758808082) [ 293.755526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.774418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.809149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 296.219672] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:48:04 (1758808084) [ 296.893378] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg223-server [ 297.563153] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:48:05 (1758808085) [ 297.596958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.613591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.642264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 297.653741] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 298.324348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[24-28]@tcp [ 298.337825] LNet: 30488:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 298.355708] LNetError: 30281:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.24@tcp is being used as a gateway but routing feature is not turned on [ 298.369094] LNetError: 30281:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.24@tcp has gone from up to down [ 299.370936] LNetError: 30281:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.28@tcp is being used as a gateway but routing feature is not turned on [ 299.381150] LNetError: 30281:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 300.391862] LNetError: 30281:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.24@tcp is being used as a gateway but routing feature is not turned on [ 300.398578] LNetError: 30281:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 301.212432] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:48:09 (1758808089) [ 301.408325] LNetError: 30489:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.27@tcp has gone from up to down [ 301.416351] LNetError: 30489:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 302.009066] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg223-server [ 302.440552] LNetError: 30281:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.28@tcp is being used as a gateway but routing feature is not turned on [ 302.447417] LNetError: 30281:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 302.757601] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:48:10 (1758808090) [ 302.788346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.456182] LNetError: 30280:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.202.24@tcp [ 304.864729] LNetError: 31416:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 304.878136] LNet: Removed LNI 192.168.202.23@tcp [ 304.888850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.920623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 304.932932] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 305.197443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[24-29/2]@tcp [ 305.208351] LNet: 31537:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 305.212488] LNet: 31537:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 305.220801] LNetError: 31426:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.24@tcp has gone from up to down [ 305.225285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[24-29/2]@tcp [ 305.226196] LNetError: 31426:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 307.997140] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:48:16 (1758808096) [ 308.061449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.075506] LNetError: 32125:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 308.085593] LNet: Removed LNI 192.168.202.23@tcp [ 308.100365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.142188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 308.183759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.214159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.251435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 308.290687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.310620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.338467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 308.373549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.392389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.426675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 308.468752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.487913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.513217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 308.546396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.563474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.592291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 308.623957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.641498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.665643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 308.703840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.721514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.751407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 308.785996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.804551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.831972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 308.871317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.891934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.928048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 308.964197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.982963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.014630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 309.045881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.065384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.096213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 311.646183] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:48:19 (1758808099) [ 312.456243] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg223-server [ 313.203713] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:48:21 (1758808101) [ 313.972143] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg223-server [ 314.651223] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:48:22 (1758808102) [ 314.996577] Key type .llcrypt unregistered [ 314.997844] Key type ._llcrypt unregistered [ 317.366241] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 317.426344] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 317.427878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 317.430063] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 319.668231] Key type ._llcrypt registered [ 319.669556] Key type .llcrypt registered [ 319.749777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 319.756071] alg: No test for adler32 (adler32-zlib) [ 320.573293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.601571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 320.613959] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 320.879817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 320.890592] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 321.707620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 321.717575] LNet: Removed LNI 10.1.2.3@tcp [ 321.769694] LNet: 34232:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.773450] LNetError: 34232:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.786435] LNet: Removed LNI 192.168.202.23@tcp [ 322.057503] Key type .llcrypt unregistered [ 322.059232] Key type ._llcrypt unregistered [ 326.641193] Key type ._llcrypt registered [ 326.642598] Key type .llcrypt registered [ 327.241578] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:48:35 (1758808115) [ 327.457422] Key type .llcrypt unregistered [ 327.459205] Key type ._llcrypt unregistered [ 329.796213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 329.835248] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 329.837730] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 329.841420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 332.129788] Key type ._llcrypt registered [ 332.131996] Key type .llcrypt registered [ 332.231347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 332.239429] alg: No test for adler32 (adler32-zlib) [ 333.055786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.083823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 333.098763] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 333.437237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 333.448690] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 334.586862] LNet: 35708:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.590471] LNetError: 35708:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 334.594247] LNet: Removed LNI 192.168.202.23@tcp [ 334.603953] LNet: Removed LNI 10.1.2.3@tcp [ 334.918385] Key type .llcrypt unregistered [ 334.919690] Key type ._llcrypt unregistered [ 339.562149] Key type ._llcrypt registered [ 339.564286] Key type .llcrypt registered [ 340.201596] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:48:48 (1758808128) [ 340.480385] Key type .llcrypt unregistered [ 340.482053] Key type ._llcrypt unregistered [ 343.068165] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 343.149952] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 343.152345] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 343.154901] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 345.438624] Key type ._llcrypt registered [ 345.440210] Key type .llcrypt registered [ 345.532888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 345.542743] alg: No test for adler32 (adler32-zlib) [ 346.371605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.399966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 346.412063] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 347.318136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 347.327720] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 348.317516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 348.333197] LNet: Removed LNI 10.1.2.3@tcp [ 348.342587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 348.365552] LNet: Removed LNI 10.1.2.31@tcp [ 348.375129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 348.393568] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 348.754605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 348.767655] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 349.255447] LNet: 37608:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.259760] LNetError: 37608:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 349.264355] LNet: Removed LNI 10.1.2.3@tcp [ 349.280597] LNet: Removed LNI 10.1.2.31@tcp [ 349.645660] Key type .llcrypt unregistered [ 349.647134] Key type ._llcrypt unregistered [ 354.537753] Key type ._llcrypt registered [ 354.540437] Key type .llcrypt registered [ 355.160869] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:49:03 (1758808143) [ 355.323039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 355.329659] alg: No test for adler32 (adler32-zlib) [ 356.146577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.176519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 356.188589] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 356.519384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.532725] LNet: 38553:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.535615] LNetError: 38553:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 356.548459] LNet: Removed LNI 192.168.202.23@tcp [ 356.558177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.584586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.23@tcp [ 356.600065] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 356.923635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.23@tcp [ 356.950494] LNet: Removed LNI 192.168.202.23@tcp [ 357.268470] Key type .llcrypt unregistered [ 357.270062] Key type ._llcrypt unregistered [ 362.099095] Key type ._llcrypt registered [ 362.100422] Key type .llcrypt registered [ 362.709801] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:49:10 (1758808150) [ 363.547106] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg223-server [ 364.279792] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:49:12 (1758808152) [ 364.507325] Key type .llcrypt unregistered [ 364.509676] Key type ._llcrypt unregistered [ 366.939677] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 366.983796] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 366.986477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 366.989291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 369.277138] Key type ._llcrypt registered [ 369.280918] Key type .llcrypt registered [ 369.370209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 369.377915] alg: No test for adler32 (adler32-zlib) [ 370.200631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.235585] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 370.537631] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 371.445798] LNet: 40507:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.449768] LNetError: 40507:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 371.453933] LNet: Removed LNI 192.168.202.23@tcp1 [ 371.466543] LNet: Removed LNI 10.1.2.3@tcp2 [ 371.742585] Key type .llcrypt unregistered [ 371.744269] Key type ._llcrypt unregistered [ 376.013818] Key type ._llcrypt registered [ 376.015236] Key type .llcrypt registered [ 376.575517] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:49:24 (1758808164) [ 376.724653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 376.731770] alg: No test for adler32 (adler32-zlib) [ 377.552833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.578069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 377.589832] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 377.936857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.947488] LNet: 41463:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.950590] LNetError: 41463:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 377.964535] LNet: Removed LNI 192.168.202.23@tcp [ 377.975582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.022548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 378.034198] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 380.841371] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:49:29 (1758808169) [ 380.906207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.919222] LNet: 42053:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.922558] LNetError: 42053:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 380.931776] LNet: Removed LNI 192.168.202.23@tcp [ 380.944159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.979819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 380.992740] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 381.316286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.23 [ 381.328934] LNet: Removed LNI 192.168.202.23@tcp [ 384.098304] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:49:32 (1758808172) [ 384.164745] LNet: 42754:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.169934] LNetError: 42754:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 384.179721] LNet: Removed LNI 192.168.202.23@tcp [ 384.491545] Key type .llcrypt unregistered [ 384.492694] Key type ._llcrypt unregistered [ 386.765850] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 386.898128] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 386.900995] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 386.904351] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 389.215659] Key type ._llcrypt registered [ 389.217045] Key type .llcrypt registered [ 389.286961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 389.293938] alg: No test for adler32 (adler32-zlib) [ 390.271627] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 392.927996] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:49:41 (1758808181) [ 392.983499] LNet: 43656:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.986941] LNetError: 43656:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 392.998194] LNet: Removed LNI 10.1.2.3@tcp [ 393.282784] Key type .llcrypt unregistered [ 393.284136] Key type ._llcrypt unregistered [ 395.541532] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 395.657220] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 395.659686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 395.662760] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 397.956514] Key type ._llcrypt registered [ 397.957752] Key type .llcrypt registered [ 398.025730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 398.032776] alg: No test for adler32 (adler32-zlib) [ 398.887707] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 401.773345] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:49:49 (1758808189) [ 401.837145] LNet: 44540:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.841489] LNetError: 44540:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 401.849812] LNet: Removed LNI 10.1.2.3@tcp [ 402.164529] Key type .llcrypt unregistered [ 402.166690] Key type ._llcrypt unregistered [ 404.531603] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 404.642393] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 404.644748] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 404.647437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 407.002629] Key type ._llcrypt registered [ 407.004136] Key type .llcrypt registered [ 407.079146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 407.087261] alg: No test for adler32 (adler32-zlib) [ 407.939295] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 410.725829] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:49:58 (1758808198) [ 410.766146] LNet: 45428:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.769659] LNetError: 45428:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 410.777793] LNet: Removed LNI 10.1.2.3@tcp [ 411.059448] Key type .llcrypt unregistered [ 411.061108] Key type ._llcrypt unregistered [ 413.394894] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 413.500604] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 413.503416] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 413.509635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 415.801599] Key type ._llcrypt registered [ 415.803052] Key type .llcrypt registered [ 415.874445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 415.880311] alg: No test for adler32 (adler32-zlib) [ 416.694916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.730896] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 420.102664] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:50:08 (1758808208) [ 420.275888] LNet: 46655:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.279558] LNetError: 46655:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 420.288054] LNet: Removed LNI 10.1.2.3@tcp [ 420.550412] Key type .llcrypt unregistered [ 420.552054] Key type ._llcrypt unregistered [ 422.770197] Key type ._llcrypt registered [ 422.771715] Key type .llcrypt registered [ 422.842427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 422.849331] alg: No test for adler32 (adler32-zlib) [ 423.668178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 423.679805] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 426.912496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 427.200269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_interrupt -r 1 [ 427.217958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_interrupt -r 1 [ 427.238716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 427.249566] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 427.380054] LNet: 47079:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 427.415974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 427.431897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 427.447903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 427.665897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_dropped -r 1 [ 427.683943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_dropped -r 1 [ 427.699770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 427.866345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 427.883544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 427.899284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 428.101982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_aborted -r 1 [ 428.120380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_aborted -r 1 [ 428.135878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 428.144696] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 428.147542] LNet: Skipped 1 previous similar message [ 428.264279] LNet: 47274:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 428.267398] LNet: 47274:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 428.299273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 428.316696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 428.333680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 428.537921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_no_route -r 1 [ 428.556987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_no_route -r 1 [ 428.573895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 428.740417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 428.756966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 428.772839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 428.992206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_timeout -r 1 [ 429.010960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_timeout -r 1 [ 429.028281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 429.202829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 429.218744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 429.232671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 429.449733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_error -r 1 [ 429.467801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_error -r 1 [ 429.486557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 429.496211] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 429.498922] LNet: Skipped 2 previous similar messages [ 429.640125] LNet: 47565:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 429.643178] LNet: 47565:0:(api-ni.c:338:recovery_interval_set()) Skipped 2 previous similar messages [ 429.672374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 429.687401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 429.702088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 429.812803] LNet: 47601:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.815802] LNetError: 47601:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 429.825562] LNet: Removed LNI 192.168.202.23@tcp [ 429.995996] Key type .llcrypt unregistered [ 429.997289] Key type ._llcrypt unregistered [ 435.960026] Key type ._llcrypt registered [ 435.961235] Key type .llcrypt registered [ 436.463917] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:50:24 (1758808224) [ 436.770247] Key type .llcrypt unregistered [ 436.771309] Key type ._llcrypt unregistered [ 439.000801] Key type ._llcrypt registered [ 439.002239] Key type .llcrypt registered [ 439.078454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 439.086618] alg: No test for adler32 (adler32-zlib) [ 439.891096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 439.903445] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 443.110564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 446.855473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.869804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 446.878784] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 447.407686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_interrupt -r 1 [ 447.425937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_interrupt -r 1 [ 447.444310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_interrupt -r 1 [ 447.462365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_interrupt -r 1 [ 447.479448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_interrupt -r 1 [ 447.495868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_interrupt -r 1 [ 447.513712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_interrupt -r 1 [ 447.530419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_interrupt -r 1 [ 447.548128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 447.556821] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 447.560333] LNet: 1 local NIs in recovery (showing 1): 192.168.202.23@tcp [ 447.673619] LNet: 49046:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 447.704459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 447.720331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 447.734557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 447.931227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_dropped -r 1 [ 447.948655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_dropped -r 1 [ 447.965701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_dropped -r 1 [ 447.983975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_dropped -r 1 [ 448.001568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_dropped -r 1 [ 448.018623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_dropped -r 1 [ 448.035742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_dropped -r 1 [ 448.053843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_dropped -r 1 [ 448.071501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 448.082495] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 448.085766] LNet: Skipped 2 previous similar messages [ 448.207255] LNet: 49155:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 448.237256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 448.251835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 448.267304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 448.460781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_aborted -r 1 [ 448.476820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_aborted -r 1 [ 448.493127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_aborted -r 1 [ 448.512173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_aborted -r 1 [ 448.529776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_aborted -r 1 [ 448.546495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_aborted -r 1 [ 448.563137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_aborted -r 1 [ 448.581617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_aborted -r 1 [ 448.599029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 448.766218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 448.782279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 448.799081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 449.017579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_no_route -r 1 [ 449.035222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_no_route -r 1 [ 449.054459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_no_route -r 1 [ 449.073387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_no_route -r 1 [ 449.091774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_no_route -r 1 [ 449.110316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_no_route -r 1 [ 449.128527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_no_route -r 1 [ 449.146344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_no_route -r 1 [ 449.161743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 449.170452] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 449.173331] LNet: Skipped 6 previous similar messages [ 449.314990] LNet: 49373:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 449.319321] LNet: 49373:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 449.356817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 449.374044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 449.388761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 449.604678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_timeout -r 1 [ 449.624256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_timeout -r 1 [ 449.643140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_timeout -r 1 [ 449.662437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_timeout -r 1 [ 449.682640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_timeout -r 1 [ 449.700216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_timeout -r 1 [ 449.719551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_timeout -r 1 [ 449.737226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_timeout -r 1 [ 449.756317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 449.927718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 449.942113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 449.956880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 450.184348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_error -r 1 [ 450.201460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_error -r 1 [ 450.220035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_error -r 1 [ 450.237065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_error -r 1 [ 450.255267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_error -r 1 [ 450.273574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_error -r 1 [ 450.291599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_error -r 1 [ 450.312386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_error -r 1 [ 450.331814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 450.499657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 450.517213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 450.534789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 452.920805] LNet: 49661:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.923771] LNetError: 49661:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 452.927154] LNet: Removed LNI 192.168.202.23@tcp [ 453.991305] LNet: Removed LNI 192.168.202.23@tcp1 [ 454.159639] Key type .llcrypt unregistered [ 454.161630] Key type ._llcrypt unregistered [ 460.244177] Key type ._llcrypt registered [ 460.245395] Key type .llcrypt registered [ 460.741877] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:50:49 (1758808249) [ 461.036264] Key type .llcrypt unregistered [ 461.037390] Key type ._llcrypt unregistered [ 463.157889] Key type ._llcrypt registered [ 463.159162] Key type .llcrypt registered [ 463.243914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 463.255220] alg: No test for adler32 (adler32-zlib) [ 464.071814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 464.086800] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 467.546436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 467.827604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e remote_dropped -r 1 [ 467.845262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e remote_dropped -r 1 [ 467.864696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 467.875068] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 468.004065] LNet: 50897:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 468.037183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 468.055122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 468.074060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 468.291461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e remote_error -r 1 [ 468.307692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e remote_error -r 1 [ 468.324737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 468.487800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 468.505321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 468.520375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 468.734661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e remote_timeout -r 1 [ 468.752321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e remote_timeout -r 1 [ 468.770300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 468.780388] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 468.783686] LNet: Skipped 1 previous similar message [ 468.904114] LNet: 51091:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 468.907787] LNet: 51091:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 468.939696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 468.956333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 468.972606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 469.086288] LNet: 51127:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.089877] LNetError: 51127:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 469.098386] LNet: Removed LNI 192.168.202.23@tcp [ 469.275439] Key type .llcrypt unregistered [ 469.277278] Key type ._llcrypt unregistered [ 475.443857] Key type ._llcrypt registered [ 475.445340] Key type .llcrypt registered [ 476.028113] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:51:04 (1758808264) [ 476.392694] Key type .llcrypt unregistered [ 476.394424] Key type ._llcrypt unregistered [ 478.822150] Key type ._llcrypt registered [ 478.823716] Key type .llcrypt registered [ 478.911428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 478.920617] alg: No test for adler32 (adler32-zlib) [ 479.730038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 479.740717] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 483.117372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 486.408025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.422660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 486.433649] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 486.976664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e remote_dropped -r 1 [ 486.994589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e remote_dropped -r 1 [ 487.010946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e remote_dropped -r 1 [ 487.028798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e remote_dropped -r 1 [ 487.049180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e remote_dropped -r 1 [ 487.067474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e remote_dropped -r 1 [ 487.085305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e remote_dropped -r 1 [ 487.101688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e remote_dropped -r 1 [ 487.121768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 487.132411] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 487.135439] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.123@tcp [ 487.257076] LNet: 52571:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 487.293493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 487.308725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 487.325857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 487.342522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 487.552908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e remote_error -r 1 [ 487.570807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e remote_error -r 1 [ 487.587738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e remote_error -r 1 [ 487.607384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e remote_error -r 1 [ 487.629910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e remote_error -r 1 [ 487.650855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e remote_error -r 1 [ 487.667809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e remote_error -r 1 [ 487.683169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e remote_error -r 1 [ 487.700878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 487.711084] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 487.713135] LNet: Skipped 2 previous similar messages [ 487.849168] LNet: 52682:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 487.882494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 487.898228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 487.914579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 487.929698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 488.148121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e remote_timeout -r 1 [ 488.166663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e remote_timeout -r 1 [ 488.184937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e remote_timeout -r 1 [ 488.201805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e remote_timeout -r 1 [ 488.221160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e remote_timeout -r 1 [ 488.238941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e remote_timeout -r 1 [ 488.256210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e remote_timeout -r 1 [ 488.275332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e remote_timeout -r 1 [ 488.293639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.123@tcp [ 488.462750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 488.479047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 488.497740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 488.514659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 490.874850] LNet: 52866:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 490.878543] LNetError: 52866:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 490.883426] LNet: Removed LNI 192.168.202.23@tcp [ 490.896695] LNet: Removed LNI 192.168.202.23@tcp1 [ 491.082525] Key type .llcrypt unregistered [ 491.084265] Key type ._llcrypt unregistered [ 497.552856] Key type ._llcrypt registered [ 497.554348] Key type .llcrypt registered [ 498.121762] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:51:26 (1758808286) [ 498.359467] Key type .llcrypt unregistered [ 498.361441] Key type ._llcrypt unregistered [ 501.080796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 501.189311] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 501.195861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 501.201340] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 503.435711] Key type ._llcrypt registered [ 503.437264] Key type .llcrypt registered [ 503.509390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 503.517242] alg: No test for adler32 (adler32-zlib) [ 504.349207] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 504.368389] LNet: 53946:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.372599] LNetError: 53946:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 504.383060] LNet: Removed LNI 192.168.202.23@tcp [ 504.677682] Key type .llcrypt unregistered [ 504.679571] Key type ._llcrypt unregistered [ 507.058190] Key type ._llcrypt registered [ 507.059788] Key type .llcrypt registered [ 507.140062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 507.150590] alg: No test for adler32 (adler32-zlib) [ 507.972489] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 507.974225] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 507.995286] LNet: 54215:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 507.999561] LNetError: 54215:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 508.002709] LNet: Removed LNI 192.168.202.23@tcp [ 508.011429] LNet: Removed LNI 10.1.2.3@tcp1 [ 508.328567] Key type .llcrypt unregistered [ 508.330464] Key type ._llcrypt unregistered [ 510.615645] Key type ._llcrypt registered [ 510.617086] Key type .llcrypt registered [ 510.705234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 510.712959] alg: No test for adler32 (adler32-zlib) [ 511.531100] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 511.533888] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 511.552376] LNet: 54483:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 511.555066] LNetError: 54483:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 511.558129] LNet: Removed LNI 192.168.202.23@tcp [ 511.567481] LNet: Removed LNI 10.1.2.3@tcp [ 511.864644] Key type .llcrypt unregistered [ 511.866228] Key type ._llcrypt unregistered [ 514.189866] Key type ._llcrypt registered [ 514.191844] Key type .llcrypt registered [ 514.294422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 514.306194] alg: No test for adler32 (adler32-zlib) [ 515.134711] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 515.137158] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 515.156303] LNet: 54753:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 515.164192] LNetError: 54753:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 515.167489] LNet: Removed LNI 192.168.202.23@tcp [ 515.174437] LNet: Removed LNI 10.1.2.3@tcp [ 515.478660] Key type .llcrypt unregistered [ 515.480566] Key type ._llcrypt unregistered [ 517.699448] Key type ._llcrypt registered [ 517.700706] Key type .llcrypt registered [ 517.775722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 517.783140] alg: No test for adler32 (adler32-zlib) [ 518.593928] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 518.595541] LNetError: 55012:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 518.598802] LNetError: Error -100 starting up LNI tcp [ 518.607360] LNet: Removed LNI 192.168.202.23@tcp [ 518.896396] Key type .llcrypt unregistered [ 518.898525] Key type ._llcrypt unregistered [ 523.126731] Key type ._llcrypt registered [ 523.128261] Key type .llcrypt registered [ 523.698364] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:51:51 (1758808311) [ 524.084373] Key type .llcrypt unregistered [ 524.086177] Key type ._llcrypt unregistered [ 526.477430] Key type ._llcrypt registered [ 526.482047] Key type .llcrypt registered [ 526.579038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 526.585995] alg: No test for adler32 (adler32-zlib) [ 527.417320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 527.433213] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 530.895338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 531.188913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e network_timeout -r 1 [ 531.207556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e network_timeout -r 1 [ 531.227293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 531.240035] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 531.384634] LNet: 56251:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 531.417789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 531.437492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 531.560532] LNet: 56285:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 531.563669] LNetError: 56285:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 531.579491] LNet: Removed LNI 192.168.202.23@tcp [ 531.769557] Key type .llcrypt unregistered [ 531.771290] Key type ._llcrypt unregistered [ 538.500232] Key type ._llcrypt registered [ 538.501369] Key type .llcrypt registered [ 538.577695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 538.584813] alg: No test for adler32 (adler32-zlib) [ 539.414121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 539.427830] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 542.742622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 546.026568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.044257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 546.055194] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 546.651916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e network_timeout -r 1 [ 546.669261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e network_timeout -r 1 [ 546.688533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e network_timeout -r 1 [ 546.708665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e network_timeout -r 1 [ 546.730490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e network_timeout -r 1 [ 546.749433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e network_timeout -r 1 [ 546.768749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e network_timeout -r 1 [ 546.788991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e network_timeout -r 1 [ 546.815441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 546.828468] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 546.961691] LNet: 57143:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 546.995284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 547.014407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 549.308965] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 549.439228] LNet: 57211:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.442632] LNetError: 57211:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.446735] LNet: Removed LNI 192.168.202.23@tcp [ 549.452367] LNet: Removed LNI 192.168.202.23@tcp1 [ 549.656672] Key type .llcrypt unregistered [ 549.658222] Key type ._llcrypt unregistered [ 555.943681] Key type ._llcrypt registered [ 555.945211] Key type .llcrypt registered [ 556.534950] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:52:24 (1758808344) [ 556.689198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 556.697064] alg: No test for adler32 (adler32-zlib) [ 557.513248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.544545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 557.557555] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 557.872777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 557.885057] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 558.244827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.23@tcp [ 558.281087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 558.303308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 558.320473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 558.337444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 558.354709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 558.374553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.23@tcp [ 558.385393] LNetError: 58299:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.202.23@tcp to recovery [ 558.560510] LNet: 1 local NIs in recovery (showing 1): 192.168.202.23@tcp [ 558.562881] LNet: There was an unexpected network error while writing to 192.168.202.23: rc = -22 [ 560.608614] LNet: There was an unexpected network error while writing to 192.168.202.23: rc = -22 [ 561.601979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 564.704386] LNet: There was an unexpected network error while writing to 192.168.202.23: rc = -22 [ 564.840809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 564.863255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 564.901265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.910388] LNet: 58407:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.914480] LNetError: 58407:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.916928] LNet: Removed LNI 192.168.202.23@tcp [ 564.923799] LNet: Removed LNI 192.168.202.23@tcp1 [ 564.936996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.964186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 564.981119] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 565.335284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 565.351270] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 565.736856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.23@tcp [ 565.755370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 565.781425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 565.806563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 565.824937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 565.844709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 565.867092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 565.886512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.23@tcp [ 565.898655] LNetError: 58673:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.202.23@tcp to recovery [ 567.008680] LNet: There was an unexpected network error while writing to 192.168.202.23: rc = -22 [ 567.012090] LNet: Skipped 1 previous similar message [ 568.098873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 571.104416] LNet: There was an unexpected network error while writing to 192.168.202.23: rc = -22 [ 575.531896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 575.549385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 575.566749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 578.261818] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:52:46 (1758808366) [ 578.325461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.338644] LNet: 59377:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.341784] LNetError: 59377:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 578.344787] LNetError: 59377:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 578.347925] LNet: Removed LNI 192.168.202.23@tcp [ 578.353709] LNet: Removed LNI 192.168.202.23@tcp1 [ 578.364331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.394058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 578.407649] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 578.714168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 578.724384] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 579.075583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.23@tcp [ 579.114648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 579.132512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 579.150763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 579.168296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 579.186199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 579.201826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.23@tcp --health 0 [ 579.242991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 579.424201] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.23@tcp [ 580.450824] LNet: There was an unexpected network error while writing to 192.168.202.23: rc = -22 [ 580.453865] LNet: Skipped 1 previous similar message [ 594.857527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 594.925215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 594.943230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 594.957798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.23@tcp --health 500 [ 595.001291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 601.254654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 601.282090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.294717] LNet: 59872:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.298940] LNetError: 59872:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.302605] LNetError: 59872:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 601.307140] LNet: Removed LNI 192.168.202.23@tcp [ 601.317513] LNet: Removed LNI 192.168.202.23@tcp1 [ 601.328208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.357356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 601.372752] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 601.735413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 601.746060] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 602.156772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.23@tcp [ 602.181480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 602.212854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 602.236482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 602.256627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 602.276479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 602.301737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 602.322477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.23@tcp --health 0 [ 603.425236] LNet: There was an unexpected network error while writing to 192.168.202.23: rc = -22 [ 603.428627] LNet: Skipped 3 previous similar messages [ 604.506019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 613.982393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 613.998707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 614.016451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 616.885637] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:53:25 (1758808405) [ 617.073717] LNet: 60921:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.076751] LNetError: 60921:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.079617] LNetError: 60921:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 617.083284] LNet: Removed LNI 192.168.202.23@tcp [ 617.093580] LNet: Removed LNI 192.168.202.23@tcp1 [ 617.376156] Key type .llcrypt unregistered [ 617.377524] Key type ._llcrypt unregistered [ 619.817524] Key type ._llcrypt registered [ 619.819061] Key type .llcrypt registered [ 619.913826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 619.921764] alg: No test for adler32 (adler32-zlib) [ 620.736027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 620.750145] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 624.690961] Lustre: DEBUG MARKER: Initial discovery [ 624.741259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.123@tcp [ 625.764259] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 625.808433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 625.828228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.123@tcp [ 625.841682] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 625.851579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 625.870286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.123@tcp -r 1 -m GET -l 3 [ 625.888135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.123@tcp -r 1 -m GET -e local_error [ 625.905616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.123@tcp -r 1 -m PUT -l 6 [ 626.506097] Lustre: DEBUG MARKER: Force oleg223-client.virtnet to discover 192.168.202.123@tcp (in background) [ 626.579391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.123@tcp [ 627.182697] Lustre: DEBUG MARKER: Force oleg223-server to discover 192.168.202.23@tcp [ 628.151311] Lustre: DEBUG MARKER: Wait for 61677 [ 629.600372] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 630.244757] Lustre: DEBUG MARKER: Finished wait on 61677 [ 630.315721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 630.335082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 630.467058] LNet: 62125:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.470928] LNetError: 62125:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.488755] LNet: Removed LNI 192.168.202.23@tcp [ 630.669793] Key type .llcrypt unregistered [ 630.671434] Key type ._llcrypt unregistered [ 637.231628] Key type ._llcrypt registered [ 637.233245] Key type .llcrypt registered [ 637.836596] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:53:45 (1758808425) [ 638.072498] Key type .llcrypt unregistered [ 638.074214] Key type ._llcrypt unregistered [ 640.410644] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 640.481737] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 640.484067] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 640.486469] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 642.722750] Key type ._llcrypt registered [ 642.724412] Key type .llcrypt registered [ 642.831838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 642.845180] alg: No test for adler32 (adler32-zlib) [ 643.671465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.705804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 643.724963] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 644.068323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 644.078688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.828213] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:53:56 (1758808436) [ 647.904524] LNet: 64184:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.908157] LNetError: 64184:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.912444] LNet: Removed LNI 192.168.202.23@tcp [ 647.919603] LNet: Removed LNI 10.1.2.3@tcp [ 648.265690] Key type .llcrypt unregistered [ 648.270493] Key type ._llcrypt unregistered [ 651.034048] Key type ._llcrypt registered [ 651.035366] Key type .llcrypt registered [ 651.131241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 651.138966] alg: No test for adler32 (adler32-zlib) [ 651.963147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.996969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 652.013723] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 652.381373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 652.394698] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 652.882123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.23@tcp [ 653.920147] LNetError: Unexpected error -22 connecting to 192.168.202.23@tcp at host 192.168.202.23:988 [ 656.167350] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:54:04 (1758808444) [ 656.238748] LNet: 65307:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.243772] LNetError: 65307:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.247996] LNet: Removed LNI 192.168.202.23@tcp [ 657.318640] LNet: Removed LNI 10.1.2.3@tcp [ 657.614875] Key type .llcrypt unregistered [ 657.616515] Key type ._llcrypt unregistered [ 660.033769] Key type ._llcrypt registered [ 660.035373] Key type .llcrypt registered [ 660.123563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 660.131898] alg: No test for adler32 (adler32-zlib) [ 660.955272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.988752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 661.011046] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 661.332877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 661.343383] LNet: Added LNI 192.168.202.23@tcp2 [8/256/0/180] [ 661.701545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.23@tcp1 --nid 192.168.202.23@tcp2 [ 663.572906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.23@tcp1 192.168.202.23@tcp2 [ 663.583469] LNetError: 66133:0:(lib-move.c:2262:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.23@tcp1 to 192.168.202.23@tcp2 - Route table may be misconfigured [ 663.599124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.23@tcp2 192.168.202.23@tcp1 [ 666.395208] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:54:14 (1758808454) [ 666.456226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.468329] LNet: 66702:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.473126] LNetError: 66702:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.476667] LNet: Removed LNI 192.168.202.23@tcp1 [ 667.494874] LNet: Removed LNI 192.168.202.23@tcp2 [ 667.507214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.536404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 667.550962] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 667.887609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 667.897444] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 668.334557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.23@tcp [ 668.359305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.23@tcp 192.168.202.23@tcp [ 668.379410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.23@tcp1 192.168.202.23@tcp1 [ 668.405573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.23@tcp -d 192.168.202.23@tcp -e network_timeout [ 668.427225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -e network_timeout [ 668.452177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -e network_timeout [ 668.476216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -e network_timeout [ 668.499448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 668.512576] LNet: There was an unexpected network error while writing to 192.168.202.23: rc = -22 [ 668.524748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 668.569118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 668.577076] LNet: 1 local NIs in recovery (showing 1): 192.168.202.23@tcp [ 668.611264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 671.428814] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:54:19 (1758808459) [ 671.492828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.507728] LNet: 67558:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.511462] LNetError: 67558:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.514537] LNetError: 67558:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 671.518248] LNet: Removed LNI 192.168.202.23@tcp [ 672.551351] LNet: Removed LNI 192.168.202.23@tcp1 [ 672.563934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.604492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 672.713449] LNet: 67597:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.908532] Key type .llcrypt unregistered [ 672.910150] Key type ._llcrypt unregistered [ 677.538635] Key type ._llcrypt registered [ 677.540229] Key type .llcrypt registered [ 678.182883] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:54:26 (1758808466) [ 678.340219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 678.347856] alg: No test for adler32 (adler32-zlib) [ 679.172600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.884892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 680.632608] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:54:28 (1758808468) [ 680.737143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.750467] LNet: 68692:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.763631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.791754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 680.806586] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 681.164391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 681.178053] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 681.612439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 681.637660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp1 [ 681.662884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.23@tcp1 [ 684.797513] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:54:32 (1758808472) [ 685.728163] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg223-server [ 686.618119] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:54:34 (1758808474) [ 687.495310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg223-server [ 688.388858] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:54:36 (1758808476) [ 689.230390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg223-server [ 689.997432] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:54:38 (1758808478) [ 690.789410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg223-server [ 691.577329] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:54:39 (1758808479) [ 692.391730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg223-server [ 693.078116] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:54:41 (1758808481) [ 693.884859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg223-server [ 694.658396] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:54:42 (1758808482) [ 695.490888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg223-server [ 696.292842] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:54:44 (1758808484) [ 697.189165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg223-server [ 697.984099] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:54:46 (1758808486) [ 698.882829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg223-server [ 699.628582] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:54:47 (1758808487) [ 699.698110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.707848] LNet: 72557:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.711493] LNetError: 72557:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.714544] LNet: Removed LNI 192.168.202.23@tcp [ 700.774615] LNet: Removed LNI 192.168.202.23@tcp1 [ 700.785383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.824778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.840506] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 701.174777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 701.212212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 701.290478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.301680] LNet: 72702:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.306232] LNetError: 72702:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.308914] LNetError: 72702:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 701.315487] LNet: Removed LNI 192.168.202.23@tcp [ 701.327373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.351904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 701.367600] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 701.750215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 701.835380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.886708] LNet: Removed LNI 192.168.202.23@tcp [ 702.896512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.943543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.958936] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 703.273633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 703.320872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 703.441245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.457511] LNet: 72988:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.461839] LNet: 72988:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 703.467198] LNetError: 72988:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.479146] LNetError: 72988:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 703.492236] LNet: Removed LNI 192.168.202.23@tcp [ 703.504569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.530589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 703.549857] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 703.914391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 704.030229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.062933] LNet: Removed LNI 192.168.202.23@tcp [ 705.074358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.102198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 705.114517] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 705.417052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 705.450820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 705.540304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.550739] LNetError: 73275:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.554014] LNetError: 73275:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 705.561514] LNet: Removed LNI 192.168.202.23@tcp [ 705.572298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.597914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 705.611694] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 705.927842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 707.090792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.134528] LNet: Removed LNI 192.168.202.23@tcp [ 708.143776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.170730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 708.185580] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 708.493592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 708.538338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 708.653355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.664156] LNet: 73570:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.666732] LNet: 73570:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 709.737330] LNet: Removed LNI 192.168.202.23@tcp [ 709.749898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.780963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 709.796597] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 710.166587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 710.269791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.284438] LNetError: 73713:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.288247] LNetError: 73713:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 710.301624] LNet: Removed LNI 192.168.202.23@tcp [ 710.315352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.343880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 710.355459] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 710.683724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 710.728599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 710.852082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.910443] LNet: Removed LNI 192.168.202.23@tcp [ 711.920468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.946578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 711.959358] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 712.327325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 712.448115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.516869] LNet: Removed LNI 192.168.202.23@tcp [ 713.544398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.584338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.602382] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 713.967716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 714.000428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 714.126777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.175053] LNet: Removed LNI 192.168.202.23@tcp [ 715.188682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.223193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 715.238126] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 715.582348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 716.794330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.808796] LNet: 74296:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.811840] LNet: 74296:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 717.862809] LNet: Removed LNI 192.168.202.23@tcp [ 717.883680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.931299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.945065] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 718.275425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 718.315533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.327822] LNetError: 74425:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.330981] LNetError: 74425:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 718.342319] LNet: Removed LNI 192.168.202.23@tcp [ 718.354140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.382739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.401727] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 718.777525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 721.821869] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:55:10 (1758808510) [ 721.896408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.914812] LNet: Removed LNI 192.168.202.23@tcp [ 721.930938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.965629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 721.985791] LNet: Added LNI 192.168.202.23@tcp231 [8/256/0/180] [ 722.642747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.661797] LNet: Removed LNI 192.168.202.23@tcp231 [ 722.673137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.711117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 722.733077] LNet: Added LNI 192.168.202.23@tcp231 [8/256/0/180] [ 722.737820] LNet: 75268:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 722.948671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 722.971654] LNet: Removed LNI 192.168.202.23@tcp231 [ 722.980048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 722.994604] LNet: Added LNI 192.168.202.23@tcp231 [8/256/0/180] [ 723.523362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.539783] LNet: Removed LNI 192.168.202.23@tcp231 [ 723.551415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.595895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 723.610922] LNet: Added LNI 192.168.202.23@tcp231 [8/256/0/0] [ 723.615584] LNet: 75417:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 723.830749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 723.851617] LNet: Removed LNI 192.168.202.23@tcp231 [ 723.861356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 723.874554] LNet: Added LNI 192.168.202.23@tcp231 [8/256/0/0] [ 724.448960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.466746] LNet: Removed LNI 192.168.202.23@tcp231 [ 724.487792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.529346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 724.544842] LNet: Added LNI 192.168.202.23@tcp231 [8/256/0/60] [ 724.793632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 724.818137] LNet: Removed LNI 192.168.202.23@tcp231 [ 724.829059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 724.849539] LNet: Added LNI 192.168.202.23@tcp231 [8/256/0/60] [ 725.423782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.444953] LNet: Removed LNI 192.168.202.23@tcp231 [ 725.455993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.490650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 725.503180] LNet: Added LNI 192.168.202.23@tcp231 [8/256/0/180] [ 725.506445] LNet: 75714:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 725.509503] LNet: 75714:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 725.714186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 725.734760] LNet: Removed LNI 192.168.202.23@tcp231 [ 725.746457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 725.761917] LNet: Added LNI 192.168.202.23@tcp231 [8/256/0/180] [ 729.126766] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:55:17 (1758808517) [ 729.197260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.221091] LNet: Removed LNI 192.168.202.23@tcp231 [ 729.233094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.264317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 729.278038] LNet: Added LNI 192.168.202.23@tcp232 [8/256/0/180] [ 729.629549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 729.650492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 729.672317] LNet: Removed LNI 192.168.202.23@tcp232 [ 729.680151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 729.701305] LNet: Added LNI 192.168.202.23@tcp232 [8/256/0/180] [ 730.056806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 730.080210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 730.110665] LNet: Removed LNI 192.168.202.23@tcp232 [ 730.123847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 730.141189] LNet: Added LNI 192.168.202.23@tcp232 [8/256/0/180] [ 730.455961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 730.473989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 730.496043] LNet: Removed LNI 192.168.202.23@tcp232 [ 730.505676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 730.520979] LNet: Added LNI 192.168.202.23@tcp232 [8/256/0/180] [ 730.873581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 730.895135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 730.913397] LNet: Removed LNI 192.168.202.23@tcp232 [ 730.924144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 733.754382] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:55:21 (1758808521) [ 733.946621] LNet: 77535:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.948851] LNet: 77535:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 734.275258] Key type .llcrypt unregistered [ 734.278403] Key type ._llcrypt unregistered [ 736.906914] Key type ._llcrypt registered [ 736.908509] Key type .llcrypt registered [ 737.040140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 737.050151] alg: No test for adler32 (adler32-zlib) [ 737.886410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 737.901509] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 741.369098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 744.824080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.850077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 744.865479] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 745.551543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_interrupt -r 2 [ 745.571426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_interrupt -r 2 [ 745.595230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_interrupt -r 2 [ 745.616440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_interrupt -r 2 [ 745.641208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_interrupt -r 2 [ 745.660820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_interrupt -r 2 [ 745.682929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_interrupt -r 2 [ 745.705101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_interrupt -r 2 [ 745.726834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.23@tcp 192.168.202.123@tcp [ 745.739217] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 745.743599] LNet: 1 local NIs in recovery (showing 1): 192.168.202.23@tcp [ 745.916941] LNet: 78180:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 745.956807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 745.978904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 746.225398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_dropped -r 2 [ 746.248124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_dropped -r 2 [ 746.272234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_dropped -r 2 [ 746.292590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_dropped -r 2 [ 746.311641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_dropped -r 2 [ 746.331293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_dropped -r 2 [ 746.354270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_dropped -r 2 [ 746.376361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_dropped -r 2 [ 746.398211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.23@tcp 192.168.202.123@tcp [ 746.410623] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 746.572980] LNet: 78288:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 746.620696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 746.638132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 746.914656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_aborted -r 2 [ 746.933797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_aborted -r 2 [ 746.964817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_aborted -r 2 [ 746.988570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_aborted -r 2 [ 747.011342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_aborted -r 2 [ 747.035395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_aborted -r 2 [ 747.054641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_aborted -r 2 [ 747.080380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_aborted -r 2 [ 747.102380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.23@tcp 192.168.202.123@tcp [ 747.353462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 747.372211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 747.657576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_no_route -r 2 [ 747.679973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_no_route -r 2 [ 747.701458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_no_route -r 2 [ 747.726906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_no_route -r 2 [ 747.748590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_no_route -r 2 [ 747.777713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_no_route -r 2 [ 747.809675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_no_route -r 2 [ 747.828326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_no_route -r 2 [ 747.849707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.23@tcp 192.168.202.123@tcp [ 747.862858] LNet: There was an unexpected network error while writing to 192.168.202.123: rc = -22 [ 747.866891] LNet: Skipped 2 previous similar messages [ 748.026886] LNet: 78505:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 748.030249] LNet: 78505:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 748.067570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 748.091832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 748.368489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e local_timeout -r 2 [ 748.387917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e local_timeout -r 2 [ 748.407124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e local_timeout -r 2 [ 748.430511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e local_timeout -r 2 [ 748.449361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e local_timeout -r 2 [ 748.471695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e local_timeout -r 2 [ 748.492188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e local_timeout -r 2 [ 748.510486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e local_timeout -r 2 [ 748.531745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.23@tcp 192.168.202.123@tcp [ 748.732689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 748.750109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 748.990242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp -m GET -e remote_dropped -r 2 [ 749.009155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.123@tcp1 -m GET -e remote_dropped -r 2 [ 749.029099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp -m GET -e remote_dropped -r 2 [ 749.047152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp -d 192.168.202.23@tcp1 -m GET -e remote_dropped -r 2 [ 749.067761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp -m GET -e remote_dropped -r 2 [ 749.087632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.123@tcp1 -m GET -e remote_dropped -r 2 [ 749.106873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp -m GET -e remote_dropped -r 2 [ 749.125993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.23@tcp1 -d 192.168.202.23@tcp1 -m GET -e remote_dropped -r 2 [ 749.145776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.23@tcp 192.168.202.123@tcp [ 749.158287] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.123@tcp [ 749.324659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 749.345645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 751.784621] LNet: 78789:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.787822] LNetError: 78789:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.791700] LNet: Removed LNI 192.168.202.23@tcp [ 751.798649] LNet: Removed LNI 192.168.202.23@tcp1 [ 751.975760] Key type .llcrypt unregistered [ 751.977065] Key type ._llcrypt unregistered [ 758.269743] Key type ._llcrypt registered [ 758.271278] Key type .llcrypt registered [ 758.856286] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:55:47 (1758808547) [ 759.017333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 759.024493] alg: No test for adler32 (adler32-zlib) [ 759.843722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.867186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 759.878961] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 760.199199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 760.229948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 760.247624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 760.281792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 760.303222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 760.324971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 763.068233] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:55:51 (1758808551) [ 763.125635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.138625] LNet: 80337:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.142372] LNetError: 80337:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.152198] LNet: Removed LNI 192.168.202.23@tcp [ 763.164648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.191021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.203914] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 766.203410] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:55:54 (1758808554) [ 766.825828] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 767.580683] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:55:55 (1758808555) [ 767.785422] LNet: 81371:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.789669] LNetError: 81371:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.800589] LNet: Removed LNI 192.168.202.23@tcp [ 768.080651] Key type .llcrypt unregistered [ 768.081643] Key type ._llcrypt unregistered [ 770.398884] Key type ._llcrypt registered [ 770.400100] Key type .llcrypt registered [ 770.496556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 770.504384] alg: No test for adler32 (adler32-zlib) [ 771.327711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 771.343592] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 774.760605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 776.577047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.123@tcp [ 776.713637] LNet: 81782:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.719657] LNetError: 81782:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.731693] LNet: Removed LNI 192.168.202.23@tcp [ 776.919572] Key type .llcrypt unregistered [ 776.921246] Key type ._llcrypt unregistered [ 781.288268] Key type ._llcrypt registered [ 781.292853] Key type .llcrypt registered [ 781.868586] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 782.523222] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 783.246620] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:56:11 (1758808571) [ 783.408937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 783.416402] alg: No test for adler32 (adler32-zlib) [ 784.233209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.269448] LNet: 82864:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.535430] Key type .llcrypt unregistered [ 784.536792] Key type ._llcrypt unregistered [ 786.742708] Key type ._llcrypt registered [ 786.744167] Key type .llcrypt registered [ 786.814942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 786.822362] alg: No test for adler32 (adler32-zlib) [ 787.641964] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 787.644144] LNet: 83116:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 787.657897] LNetError: 83128:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.24@tcp is being used as a gateway but routing feature is not turned on [ 787.665985] LNetError: 83128:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.202.24@tcp has gone from up to down [ 787.862193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.944161] LNetError: 83128:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.202.26@tcp has gone from up to down [ 790.948061] LNetError: 83128:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 790.951596] LNet: 83174:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.955155] LNetError: 83174:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.965517] LNet: Removed LNI 192.168.202.23@tcp [ 790.975376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.002595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 791.015399] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 791.018131] LNet: 83182:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 791.021424] LNet: 83182:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 791.027171] LNetError: 83182:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.24@tcp is a gateway. Can not delete it [ 791.031252] LNetError: 83182:0:(peer.c:2201:lnet_del_peer_ni()) 192.168.202.24@tcp is a router. Can not be deleted [ 791.036684] LNetError: 83179:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.25@tcp is being used as a gateway but routing feature is not turned on [ 791.043953] LNetError: 83179:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 791.048424] LNet: 83182:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 793.611365] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:56:21 (1758808581) [ 793.824428] LNetError: 83183:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.202.27@tcp has gone from up to down [ 793.828813] LNetError: 83183:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 794.338857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg223-server [ 794.973629] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:56:23 (1758808583) [ 795.685631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg223-server [ 796.355187] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:56:24 (1758808584) [ 797.000162] Lustre: Lustre: Build Version: 2.16.58_52_ga6ec4c6 [ 798.680190] Key type lgssc registered [ 799.102277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 807.700720] Key type lgssc unregistered [ 807.864170] LNet: 85597:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.867223] LNetError: 85597:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.877542] LNet: Removed LNI 192.168.202.23@tcp [ 808.126707] Key type .llcrypt unregistered [ 808.127785] Key type ._llcrypt unregistered [ 813.639429] Key type ._llcrypt registered [ 813.640735] Key type .llcrypt registered [ 814.191594] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:56:42 (1758808602) [ 814.337136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.344303] alg: No test for adler32 (adler32-zlib) [ 815.159169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.189871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 815.213138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 815.233712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 815.254898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 817.652598] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:56:45 (1758808605) [ 817.714112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.726923] LNet: 86996:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.739675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.768250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.783250] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 818.150059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.159579] LNetError: 87139:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.171569] LNet: Removed LNI 192.168.202.23@tcp [ 818.180669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.206072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 818.218889] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 818.407511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.416832] LNet: 87167:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.419504] LNet: 87167:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 818.436422] LNet: Removed LNI 192.168.202.23@tcp [ 818.445609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.471851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 818.484946] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 821.078189] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:56:49 (1758808609) [ 821.133925] LNet: 87754:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.138328] LNetError: 87754:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.142286] LNetError: 87754:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 821.152153] LNet: Removed LNI 192.168.202.23@tcp [ 821.437592] Key type .llcrypt unregistered [ 821.439339] Key type ._llcrypt unregistered [ 823.595340] Key type ._llcrypt registered [ 823.597061] Key type .llcrypt registered [ 823.670467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.676472] alg: No test for adler32 (adler32-zlib) [ 824.492294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.532185] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 824.533817] LNet: *** cfs_fail_loc=e003, val=100*** [ 824.815843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 824.828217] Ping reply received with mapping: [ 824.829808] found metadata reply NI[192.168.202.23@tcp] [ 824.831611] 17 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 824.833462] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 824.834911] 5a 5a 5a 5a 5a 5a 5a 5a [ 824.882385] LNet: 88146:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.885564] LNetError: 88146:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.900399] LNet: Removed LNI 192.168.202.23@tcp [ 825.153443] Key type .llcrypt unregistered [ 825.154931] Key type ._llcrypt unregistered [ 829.330892] Key type ._llcrypt registered [ 829.331886] Key type .llcrypt registered [ 829.874643] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:56:58 (1758808618) [ 830.068297] Key type .llcrypt unregistered [ 830.069730] Key type ._llcrypt unregistered [ 832.204049] Key type ._llcrypt registered [ 832.205246] Key type .llcrypt registered [ 832.278243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.284523] alg: No test for adler32 (adler32-zlib) [ 833.173728] LNetError: 89194:0:(api-ni.c:2709:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 833.337644] Key type .llcrypt unregistered [ 833.338838] Key type ._llcrypt unregistered [ 835.283726] Key type ._llcrypt registered [ 835.284933] Key type .llcrypt registered [ 835.842886] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:57:04 (1758808624) [ 836.056539] Key type .llcrypt unregistered [ 836.058748] Key type ._llcrypt unregistered [ 838.217446] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 838.219447] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 840.401761] Key type ._llcrypt registered [ 840.403040] Key type .llcrypt registered [ 840.477972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.485649] alg: No test for adler32 (adler32-zlib) [ 841.291809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.317159] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 845.541914] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:57:13 (1758808633) [ 845.578962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.590574] LNet: 90957:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.594381] LNetError: 90957:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 845.604148] LNet: Removed LNI 10.1.2.3@tcp [ 845.617219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.961637] Key type .llcrypt unregistered [ 845.963060] Key type ._llcrypt unregistered [ 848.118518] Key type ._llcrypt registered [ 848.120132] Key type .llcrypt registered [ 848.191142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 848.198506] alg: No test for adler32 (adler32-zlib) [ 849.024402] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 849.052113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 849.111803] LNet: 91245:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.115362] LNetError: 91245:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.128387] LNet: Removed LNI 192.168.202.23@tcp [ 849.397693] Key type .llcrypt unregistered [ 849.399441] Key type ._llcrypt unregistered [ 853.359785] Key type ._llcrypt registered [ 853.361148] Key type .llcrypt registered [ 853.878219] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:57:22 (1758808642) [ 854.096298] Key type .llcrypt unregistered [ 854.097869] Key type ._llcrypt unregistered [ 856.117623] Key type ._llcrypt registered [ 856.118614] Key type .llcrypt registered [ 856.187241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.193540] alg: No test for adler32 (adler32-zlib) [ 859.296709] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:57:27 (1758808647) [ 859.353719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.375618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.403533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.414862] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 859.706916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.720401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 859.737620] LNet: Removed LNI 192.168.202.23@tcp [ 859.836879] LNet: 92975:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.011523] Key type .llcrypt unregistered [ 860.012802] Key type ._llcrypt unregistered [ 863.965864] Key type ._llcrypt registered [ 863.967287] Key type .llcrypt registered [ 864.473929] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:57:32 (1758808652) [ 864.632360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.639174] alg: No test for adler32 (adler32-zlib) [ 865.451883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.474707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.487934] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 865.776294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 865.792817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.23@tcp [ 865.801886] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 865.804218] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 865.806173] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 865, 0, 0 [ 865.808633] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 865.809852] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 865.811730] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 865.813750] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 865, 0, 0 [ 865.815799] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 865.817176] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 865.818975] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 865.820498] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 865, 0, 915 [ 865.822267] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 865.823417] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 865.825494] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 865.827328] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 865, 0, 915 [ 865.830381] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 868.141328] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:57:36 (1758808656) [ 868.306726] LNet: 94507:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.309456] LNetError: 94507:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.316517] LNet: Removed LNI 192.168.202.23@tcp [ 868.546431] Key type .llcrypt unregistered [ 868.547828] Key type ._llcrypt unregistered [ 870.548602] Key type ._llcrypt registered [ 870.549985] Key type .llcrypt registered [ 870.624795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.631348] alg: No test for adler32 (adler32-zlib) [ 871.440986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 871.451471] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 874.332128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 876.978107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.991382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 877.000517] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 877.331210] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 877.370827] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 877.373275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 877.375766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 879.508743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 879.516689] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 883.278214] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:57:51 (1758808671) [ 883.314708] LNet: 95995:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.317777] LNetError: 95995:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.321184] LNet: Removed LNI 192.168.202.23@tcp [ 884.384194] LNet: Removed LNI 192.168.202.23@tcp1 [ 884.386085] LNetError: 95995:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.388887] LNetError: 95995:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 884.397719] LNet: Removed LNI 10.1.2.3@tcp99 [ 884.640501] Key type .llcrypt unregistered [ 884.642038] Key type ._llcrypt unregistered [ 886.622934] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 886.625516] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 888.792290] Key type ._llcrypt registered [ 888.794223] Key type .llcrypt registered [ 888.875212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 888.882721] alg: No test for adler32 (adler32-zlib) [ 889.690901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.714502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 889.723570] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 889.985935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 889.997422] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 890.787258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 890.803139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 890.821125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 890.831721] LNetError: 96735:0:(peer.c:2209:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 890.846976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 890.869321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.23@tcp [ 891.051188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 891.080961] LNet: 96755:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.083386] LNetError: 96755:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.085807] LNet: Removed LNI 192.168.202.23@tcp [ 891.097637] LNet: Removed LNI 10.1.2.3@tcp [ 891.353584] Key type .llcrypt unregistered [ 891.355129] Key type ._llcrypt unregistered [ 895.039738] Key type ._llcrypt registered [ 895.040927] Key type .llcrypt registered [ 895.531330] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:58:03 (1758808683) [ 895.659422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 895.665797] alg: No test for adler32 (adler32-zlib) [ 896.468487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.489389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.501547] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 896.748857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.23@tcp [ 896.776703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg223-client@tcp [ 898.876539] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:58:07 (1758808687) [ 898.920028] LNet: 98272:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.922611] LNetError: 98272:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.929634] LNet: Removed LNI 192.168.202.23@tcp [ 899.198562] Key type .llcrypt unregistered [ 899.200132] Key type ._llcrypt unregistered [ 901.277557] Key type ._llcrypt registered [ 901.278990] Key type .llcrypt registered [ 901.343583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 901.350496] alg: No test for adler32 (adler32-zlib) [ 902.152517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 902.163331] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 902.177769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 904.419222] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:58:12 (1758808692) [ 904.944827] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 905.530190] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:58:13 (1758808693) [ 906.057679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 906.657882] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:58:14 (1758808694) [ 906.695152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.704498] LNet: 99690:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.707903] LNetError: 99690:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.715422] LNet: Removed LNI 192.168.202.23@tcp [ 906.726508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.746529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 906.756335] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 907.019175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 907.039489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 909.399736] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:58:17 (1758808697) [ 909.442717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.452397] LNet: 100397:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.455459] LNetError: 100397:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.466197] LNet: Removed LNI 192.168.202.23@tcp [ 909.475038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.498388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 909.513790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 909.771404] Key type .llcrypt unregistered [ 909.772740] Key type ._llcrypt unregistered [ 913.642658] Key type ._llcrypt registered [ 913.644577] Key type .llcrypt registered [ 914.122643] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:58:22 (1758808702) [ 914.240106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 914.245375] alg: No test for adler32 (adler32-zlib) [ 915.045917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.065365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 915.073572] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 915.299255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 915.318544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.23@tcp [ 917.436064] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:58:25 (1758808705) [ 917.466818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.476096] LNet: 101939:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.478931] LNetError: 101939:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.488823] LNet: Removed LNI 192.168.202.23@tcp [ 917.499535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.522767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 917.535257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 919.628225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 920.180529] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:58:28 (1758808708) [ 920.214373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.223047] LNet: 102660:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.232613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.336402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 920.345941] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 920.599921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 920.608136] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 920.908257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.23@tcp [ 920.927505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 920.951802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.23@tcp -m PUT -e local_timeout -r 1 [ 920.978018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 921.001262] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 921.003271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 921.094128] LNetError: 102697:0:(lib-move.c:4945:lnet_parse()) 192.168.202.23@tcp, src 192.168.202.23@tcp: Dropping PUT (error -108 looking up sender) [ 922.144387] LNet: 102972:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.147784] LNetError: 102972:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.151230] LNet: Removed LNI 192.168.202.23@tcp [ 922.159275] LNet: Removed LNI 10.1.2.3@tcp [ 922.312641] Key type .llcrypt unregistered [ 922.313898] Key type ._llcrypt unregistered [ 926.013581] Key type ._llcrypt registered [ 926.014991] Key type .llcrypt registered [ 926.497383] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:58:34 (1758808714) [ 926.616927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.623533] alg: No test for adler32 (adler32-zlib) [ 927.425921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.552768] LNet: 103820:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.781669] Key type .llcrypt unregistered [ 927.782657] Key type ._llcrypt unregistered [ 929.692964] Key type ._llcrypt registered [ 929.694129] Key type .llcrypt registered [ 929.759919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 929.765298] alg: No test for adler32 (adler32-zlib) [ 930.569284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 930.579411] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 933.388809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 933.454763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 933.667047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.123@tcp [ 933.795168] LNet: 104256:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 933.826698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 933.842176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 934.076329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 934.231697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 934.247392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 934.499093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.123@tcp [ 934.659628] LNet: 104466:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 934.662275] LNet: 104466:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 934.688236] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.123@tcp [ 934.697420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 934.715366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 934.967395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 935.116977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 935.132300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 935.242349] LNet: 104612:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.244646] LNetError: 104612:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.256326] LNet: Removed LNI 192.168.202.23@tcp [ 935.412586] Key type .llcrypt unregistered [ 935.414215] Key type ._llcrypt unregistered [ 940.788440] Key type ._llcrypt registered [ 940.789371] Key type .llcrypt registered [ 941.247762] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:58:49 (1758808729) [ 941.363923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 941.369832] alg: No test for adler32 (adler32-zlib) [ 942.173673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.330420] LNet: 105494:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.587577] Key type .llcrypt unregistered [ 942.589158] Key type ._llcrypt unregistered [ 944.741408] Key type ._llcrypt registered [ 944.743527] Key type .llcrypt registered [ 944.828695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 944.835814] alg: No test for adler32 (adler32-zlib) [ 945.641656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 945.652443] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 948.578536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.123@tcp [ 951.214537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.227182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 951.235838] LNet: Added LNI 192.168.202.23@tcp1 [8/256/0/180] [ 951.557683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 951.747150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.123@tcp [ 951.776255] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.123@tcp [ 951.851661] LNet: 106128:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 951.876560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 951.888101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 952.089681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 952.215633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 952.228554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 952.458813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.123@tcp [ 952.573273] LNet: 106338:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 952.576116] LNet: 106338:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 952.603547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 952.617421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 952.841269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 952.979035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 952.991572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 955.232310] LNet: 106519:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.235715] LNetError: 106519:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.238651] LNet: Removed LNI 192.168.202.23@tcp [ 955.244382] LNet: Removed LNI 192.168.202.23@tcp1 [ 955.396572] Key type .llcrypt unregistered [ 955.397924] Key type ._llcrypt unregistered [ 960.794806] Key type ._llcrypt registered [ 960.795890] Key type .llcrypt registered [ 961.305901] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 808 sec =========== 09:59:09 (1758808749) [ 961.543668] Key type .llcrypt unregistered [ 961.545634] Key type ._llcrypt unregistered [ 965.578169] Key type ._llcrypt registered [ 965.579240] Key type .llcrypt registered [ 965.758812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 965.765789] alg: No test for adler32 (adler32-zlib) [ 966.634625] Lustre: Lustre: Build Version: 2.16.58_52_ga6ec4c6 [ 966.734954] LNet: Added LNI 192.168.202.23@tcp [8/256/0/180] [ 968.336157] Key type lgssc registered [ 968.754098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.857781] Lustre: Mounted lustre-client [ 994.317398] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1001.761334] LustreError: 109731:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffffa0e142e76800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1001.768756] LustreError: 109731:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1001.784692] Lustre: Unmounted lustre-client [ 1013.758596] Key type lgssc unregistered [ 1013.875481] LNet: 110212:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.877991] LNetError: 110212:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.884791] LNet: Removed LNI 192.168.202.23@tcp [ 1014.164117] Key type .llcrypt unregistered [ 1014.165500] Key type ._llcrypt unregistered