[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498024110 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002361] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009138] LSM: Security Framework initializing [ 0.010046] Yama: becoming mindful. [ 0.011035] SELinux: Initializing. [ 0.012063] *** VALIDATE selinux *** [ 0.020867] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024775] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027111] *** VALIDATE tmpfs *** [ 0.028467] *** VALIDATE proc *** [ 0.029236] *** VALIDATE cgroup *** [ 0.030011] *** VALIDATE cgroup2 *** [ 0.031260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033110] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035027] Spectre V2 : User space: Vulnerable [ 0.036008] Speculative Store Bypass: Vulnerable [ 0.038922] debug: unmapping init [mem 0xffffffffa3e59000-0xffffffffa3e60fff] [ 0.040158] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041696] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043012] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.045013] ... value mask: 0000ffffffffffff [ 0.046024] ... max period: 00007fffffffffff [ 0.047010] ... fixed-purpose events: 3 [ 0.048008] ... event mask: 000000070000000f [ 0.050085] rcu: Hierarchical SRCU implementation. [ 0.052393] smp: Bringing up secondary CPUs ... [ 0.053480] x86: Booting SMP configuration: [ 0.054018] .... node #0, CPUs: #1 #2 #3 [ 0.061207] smp: Brought up 1 node, 4 CPUs [ 0.063016] smpboot: Max logical packages: 1 [ 0.064012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148245] node 0 deferred pages initialised in 81ms [ 0.155288] devtmpfs: initialized [ 0.156000] x86/mm: Memory block size: 128MB [ 0.156000] gcov: version magic: 0x41383552 [ 0.162922] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167309] pinctrl core: initialized pinctrl subsystem [ 0.169166] [ 0.169767] ************************************************************* [ 0.172014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175015] ** ** [ 0.177013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180014] ** ** [ 0.182011] ** This means that this kernel is built to expose internal ** [ 0.184012] ** IOMMU data structures, which may compromise security on ** [ 0.187014] ** your system. ** [ 0.189012] ** ** [ 0.192017] ** If you see this message and you are not debugging the ** [ 0.194017] ** kernel, report this immediately to your vendor! ** [ 0.196014] ** ** [ 0.198012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200013] ************************************************************* [ 0.202760] NET: Registered protocol family 16 [ 0.204487] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.210070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211111] cpuidle: using governor menu [ 0.213283] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214515] PCI: Using configuration type 1 for base access [ 0.215000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216086] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.219178] cryptd: max_cpu_qlen set to 1000 [ 0.221205] ACPI: Added _OSI(Module Device) [ 0.223042] ACPI: Added _OSI(Processor Device) [ 0.224049] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.229670] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.236572] ACPI: Interpreter enabled [ 0.237000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237000] ACPI: Using IOAPIC for interrupt routing [ 0.239139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.242411] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253674] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.258020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.267263] acpiphp: Slot [2] registered [ 0.269146] acpiphp: Slot [3] registered [ 0.270233] acpiphp: Slot [4] registered [ 0.273000] acpiphp: Slot [5] registered [ 0.275229] acpiphp: Slot [6] registered [ 0.277208] acpiphp: Slot [7] registered [ 0.279733] acpiphp: Slot [8] registered [ 0.282175] acpiphp: Slot [9] registered [ 0.283173] acpiphp: Slot [10] registered [ 0.286190] acpiphp: Slot [11] registered [ 0.288152] acpiphp: Slot [12] registered [ 0.289147] acpiphp: Slot [13] registered [ 0.291149] acpiphp: Slot [14] registered [ 0.293151] acpiphp: Slot [15] registered [ 0.294155] acpiphp: Slot [16] registered [ 0.296127] acpiphp: Slot [17] registered [ 0.298172] acpiphp: Slot [18] registered [ 0.299141] acpiphp: Slot [19] registered [ 0.301185] acpiphp: Slot [20] registered [ 0.303145] acpiphp: Slot [21] registered [ 0.304160] acpiphp: Slot [22] registered [ 0.306183] acpiphp: Slot [23] registered [ 0.308124] acpiphp: Slot [24] registered [ 0.309141] acpiphp: Slot [25] registered [ 0.311169] acpiphp: Slot [26] registered [ 0.313134] acpiphp: Slot [27] registered [ 0.315159] acpiphp: Slot [28] registered [ 0.316195] acpiphp: Slot [29] registered [ 0.318124] acpiphp: Slot [30] registered [ 0.319140] acpiphp: Slot [31] registered [ 0.321118] PCI host bridge to bus 0000:00 [ 0.322024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328046] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330032] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.331050] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333205] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.339128] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345068] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.351530] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.358060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361039] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.363030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.368601] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.371880] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377715] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.382019] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.391018] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.396023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.400000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.425016] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.434781] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.449000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.465020] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.477573] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.479448] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.482433] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.484419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.487243] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.493067] iommu: Default domain type: Passthrough [ 0.495518] SCSI subsystem initialized [ 0.496147] ACPI: bus type USB registered [ 0.498113] usbcore: registered new interface driver usbfs [ 0.499075] usbcore: registered new interface driver hub [ 0.501107] usbcore: registered new device driver usb [ 0.503219] pps_core: LinuxPPS API ver. 1 registered [ 0.504011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.507286] PTP clock support registered [ 0.509184] EDAC MC: Ver: 3.0.0 [ 0.511229] PCI: Using ACPI for IRQ routing [ 0.512000] NetLabel: Initializing [ 0.512023] NetLabel: domain hash size = 128 [ 0.513000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.516113] NetLabel: unlabeled traffic allowed by default [ 0.518193] vgaarb: loaded [ 0.520320] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.522020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.530000] clocksource: Switched to clocksource kvm-clock [ 0.651157] VFS: Disk quotas dquot_6.6.0 [ 0.652766] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.655512] *** VALIDATE ramfs *** [ 0.656847] *** VALIDATE hugetlbfs *** [ 0.658484] pnp: PnP ACPI init [ 0.661158] pnp: PnP ACPI: found 6 devices [ 0.678444] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.681670] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.683930] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.686191] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.688731] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.691268] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.694176] NET: Registered protocol family 2 [ 0.696714] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.701854] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.705685] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.712035] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.715709] TCP: Hash tables configured (established 65536 bind 65536) [ 0.718692] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.721838] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.724867] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.727986] NET: Registered protocol family 1 [ 0.732285] RPC: Registered named UNIX socket transport module. [ 0.734394] RPC: Registered udp transport module. [ 0.737264] RPC: Registered tcp transport module. [ 0.741139] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.743544] NET: Registered protocol family 44 [ 0.744776] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.747397] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.749961] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.752776] PCI: CLS 0 bytes, default 64 [ 0.754715] Unpacking initramfs... [ 2.232623] debug: unmapping init [mem 0xffff90d4fcc64000-0xffff90d4fffcffff] [ 2.236707] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.239101] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.242099] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.780026] Initialise system trusted keyrings [ 2.781514] Key type blacklist registered [ 2.783147] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.795701] zbud: loaded [ 2.801306] *** VALIDATE nfs *** [ 2.802420] *** VALIDATE nfs4 *** [ 2.804927] pstore: using deflate compression [ 2.808781] Platform Keyring initialized [ 2.947699] NET: Registered protocol family 38 [ 2.949273] Key type asymmetric registered [ 2.950732] Asymmetric key parser 'x509' registered [ 2.952393] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.955079] io scheduler mq-deadline registered [ 2.956517] io scheduler kyber registered [ 2.957976] io scheduler bfq registered [ 2.967401] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.970732] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.973120] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.975615] ACPI: Power Button [PWRF] [ 3.102541] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.201248] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.305337] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.332289] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.363381] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.368608] Non-volatile memory driver v1.3 [ 3.370461] Linux agpgart interface v0.103 [ 3.406234] virtio_blk virtio1: [vda] 132416 512-byte logical blocks (67.8 MB/64.7 MiB) [ 3.409472] vda: detected capacity change from 0 to 67796992 [ 3.426249] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.429239] vdb: detected capacity change from 0 to 1073741824 [ 3.437061] libphy: Fixed MDIO Bus: probed [ 3.448359] usbcore: registered new interface driver usbserial_generic [ 3.450860] usbserial: USB Serial support registered for generic [ 3.452514] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.456045] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.458173] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.461657] mousedev: PS/2 mouse device common for all mice [ 3.466304] rtc_cmos 00:05: RTC can wake from S4 [ 3.468686] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.473299] rtc_cmos 00:05: registered as rtc0 [ 3.475891] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.479924] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.481698] intel_pstate: CPU model not supported [ 3.485448] hid: raw HID events driver (C) Jiri Kosina [ 3.493053] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.495756] usbcore: registered new interface driver usbhid [ 3.495766] usbhid: USB HID core driver [ 3.498661] drop_monitor: Initializing network drop monitor service [ 3.507241] Initializing XFRM netlink socket [ 3.509330] NET: Registered protocol family 10 [ 3.514239] Segment Routing with IPv6 [ 3.515612] NET: Registered protocol family 17 [ 3.517670] mpls_gso: MPLS GSO support [ 3.522466] RAS: Correctable Errors collector initialized. [ 3.524575] AVX version of gcm_enc/dec engaged. [ 3.526312] AES CTR mode by8 optimization enabled [ 3.650728] sched_clock: Marking stable (3650701320, 0)->(4623513643, -972812323) [ 3.656122] registered taskstats version 1 [ 3.658141] Loading compiled-in X.509 certificates [ 3.660177] zswap: loaded using pool lzo/zbud [ 3.694859] Key type big_key registered [ 3.711724] Key type encrypted registered [ 3.713112] ima: No TPM chip found, activating TPM-bypass! [ 3.714844] ima: Allocated hash algorithm: sha1 [ 3.717614] ima: No architecture policies found [ 3.719541] evm: Initialising EVM extended attributes: [ 3.721352] evm: security.selinux [ 3.722600] evm: security.ima [ 3.723709] evm: security.capability [ 3.725231] evm: HMAC attrs: 0x1 [ 3.727746] rtc_cmos 00:05: setting system clock to 2025-07-25 10:40:00 UTC (1753440000) [ 3.734979] debug: unmapping init [mem 0xffffffffa4e03000-0xffffffffa4ffffff] [ 3.738514] debug: unmapping init [mem 0xffffffffa3b82000-0xffffffffa3e58fff] [ 3.747124] Write protecting the kernel read-only data: 28672k [ 3.751326] debug: unmapping init [mem 0xffffffffa2203000-0xffffffffa23fffff] [ 3.754335] debug: unmapping init [mem 0xffffffffa2b14000-0xffffffffa2bfffff] [ 3.793043] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.803725] systemd[1]: Detected virtualization kvm. [ 3.805714] systemd[1]: Detected architecture x86-64. [ 3.807971] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.836449] systemd[1]: No hostname configured. [ 3.838422] systemd[1]: Set hostname to . [ 3.840892] random: systemd: uninitialized urandom read (16 bytes read) [ 3.843772] systemd[1]: Initializing machine ID from random generator. [ 3.912601] random: ln: uninitialized urandom read (6 bytes read) [ 4.024889] random: systemd: uninitialized urandom read (16 bytes read) [ 4.028345] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.044138] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 4.058538] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.795472] device-mapper: uevent: version 1.0.3 [ 4.798681] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System.[ 5.540291] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.752687] virtio_net virtio0 ens2: renamed from eth0 [ 5.883676] scsi host0: ata_piix [ 5.910206] scsi host1: ata_piix [ 5.911943] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.914275] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.751340] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.231233] random: crng init done [ 10.232678] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.094760] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.440239] printk: systemd: 21 output lines suppressed due to ratelimiting [ 12.783577] SELinux: Disabled at runtime. [ 12.848976] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.857218] systemd[1]: Detected virtualization kvm. [ 12.861229] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.535581] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.540075] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.546712] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.550945] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.554741] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.563515] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.567565] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK [ 13.670922] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 14.203957] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.095823] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.199449] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.723927] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.725688] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit)[ 20.063082] hrtimer: interrupt took 3368862 ns [ 20.089929] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (6s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit)[ 20.935519] NFS: Registering the id_resolver key type [ 20.937668] Key type id_resolver registered [ 20.940266] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg228-client login: [ 94.785632] libcfs: loading out-of-tree module taints kernel. [ 94.861773] Key type ._llcrypt registered [ 94.879808] Key type .llcrypt registered [ 95.751000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 95.788731] alg: No test for adler32 (adler32-zlib) [ 97.446665] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 98.945968] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 98.948983] LNet: Accept secure, port 988 [ 100.842289] Key type lgssc registered [ 103.255398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 208.689223] Lustre: Mounted lustre-client [ 212.904789] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.853413] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing check_logdir /tmp/testlogs/ [ 230.786065] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing yml_node [ 234.464942] Lustre: lustre-OST0000-osc-ffff90d558430800: disconnect after 24s idle [ 234.660332] Lustre: DEBUG MARKER: Client: 2.16.56.57 [ 236.531345] Lustre: DEBUG MARKER: MDS: 2.16.56.57 [ 238.522872] Lustre: DEBUG MARKER: OSS: 2.16.56.57 [ 239.674645] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jul 25 06:43:55 EDT 2025 [ 252.479713] LustreError: 7891:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff90d558430800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 252.493938] LustreError: 7891:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 252.528228] Lustre: Unmounted lustre-client [ 275.841189] Key type lgssc unregistered [ 276.105601] LNet: 8337:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 277.164272] LNet: Removed LNI 192.168.202.28@tcp [ 278.089957] Key type .llcrypt unregistered [ 278.091986] Key type ._llcrypt unregistered [ 299.308685] Key type ._llcrypt registered [ 299.311308] Key type .llcrypt registered [ 299.469926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 299.488236] alg: No test for adler32 (adler32-zlib) [ 300.479362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 300.596122] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 300.600094] LNet: Accept secure, port 988 [ 300.637545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 304.359856] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 304.508372] LNet: 9203:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.521480] LNet: Removed LNI 192.168.202.28@tcp [ 305.080453] Key type .llcrypt unregistered [ 305.082271] Key type ._llcrypt unregistered [ 310.047807] Key type ._llcrypt registered [ 310.054299] Key type .llcrypt registered [ 311.272140] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:45:06 (1753440306) [ 311.638435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 311.654342] alg: No test for adler32 (adler32-zlib) [ 312.585393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.679932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.711388] LNet: 9614:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.749128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.831481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 312.846608] LNet: 9621:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 318.688194] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:45:14 (1753440314) [ 318.755977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.782759] LNet: 10192:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.804987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.879244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 324.012492] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:45:19 (1753440319) [ 324.141944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.158605] LNet: 10778:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.187721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.272411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 329.461644] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:45:25 (1753440325) [ 329.631412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.656742] LNet: 11366:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.719198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.767397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 334.971606] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:45:30 (1753440330) [ 335.123863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.150237] LNet: 11952:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.170320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.225766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 335.533866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 340.922903] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:45:36 (1753440336) [ 340.983770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.013761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.140344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 341.611798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 341.935573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 346.913660] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:45:42 (1753440342) [ 346.975217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.000430] LNet: 13134:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.013345] LNet: 13134:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 347.029325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.096301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 352.072912] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:45:47 (1753440347) [ 352.139087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.191369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.253299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 352.272816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 352.568571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 352.603447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 352.928492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 352.955831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 353.293426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 353.336638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 353.633324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 353.664769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 354.048862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 354.090447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 354.433295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 354.462582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 354.835117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 354.888022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 355.158643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 355.189266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 355.473147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 355.505730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 355.768787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 355.798188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 356.144950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 356.172950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 356.437597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 356.463403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 360.691784] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:45:56 (1753440356) [ 360.821923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.853621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.893919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 360.917197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 364.828280] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:46:00 (1753440360) [ 364.906301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.929776] LNet: 14988:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.932290] LNet: 14988:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 364.951145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.003697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 365.031282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 368.843047] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:46:04 (1753440364) [ 368.921971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.948837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.007336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 369.026940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 373.036856] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:46:08 (1753440368) [ 373.123558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.153702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.191187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 373.210495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 377.270097] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:46:12 (1753440372) [ 377.394600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.451418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.506693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 377.538162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 381.847454] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:46:17 (1753440377) [ 381.922533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.969736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.034189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 382.056528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 386.026770] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:46:22 (1753440382) [ 386.091565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.112653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.164924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 386.378170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 390.477744] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:46:26 (1753440386) [ 390.559639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.589390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.728356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 390.872817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 394.953104] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:46:30 (1753440390) [ 395.065359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.103990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.189787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 395.320800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 399.665116] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:46:35 (1753440395) [ 399.740325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.758348] LNet: 19743:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.763699] LNet: 19743:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 399.784566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.156101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 400.361331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 405.319439] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:46:40 (1753440400) [ 405.387910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.442880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.508471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 405.558594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 410.189673] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:46:46 (1753440406) [ 410.287795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.344135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.397849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 410.434581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 414.947954] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:46:50 (1753440410) [ 415.120250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.141891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.190121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 419.489260] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:46:55 (1753440415) [ 419.567310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.610701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.663085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 423.705836] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:46:59 (1753440419) [ 423.771523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.797078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.859954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 424.075762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 427.996768] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:47:03 (1753440423) [ 428.138335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.176567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.236502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 428.274169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 432.604417] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:47:08 (1753440428) [ 432.673888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.696104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.735349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 432.756456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 437.268731] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:47:13 (1753440433) [ 437.369098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.406782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.454208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 437.475960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 441.816902] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:47:17 (1753440437) [ 441.893906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.916568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.966079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 441.995829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 446.364481] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:47:22 (1753440442) [ 446.431339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.478328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.537325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 446.610615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 446.623153] LNetError: 25716:0:(peer.c:2185:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 446.655323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 446.681723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 451.106812] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:47:26 (1753440446) [ 451.186447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.223839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.279884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.we5Fr3SxMo [ 451.350674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.we5Fr3SxMo [ 451.382871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.we5Fr3SxMo [ 455.406291] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:47:31 (1753440451) [ 455.479854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.509548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.558233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 455.579254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 455.605937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 459.335875] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:47:35 (1753440455) [ 459.420303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.454052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.512224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 459.539696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 459.577763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 459.609470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 459.640778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 459.664805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 459.694304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 459.758683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 459.781276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 459.806847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 459.834450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 459.870825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 459.904508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 459.924937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 459.947914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 459.970597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 459.991244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 460.017784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 460.046219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 460.060402] LNetError: 27515:0:(router.c:632:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 464.007235] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:47:39 (1753440459) [ 464.066034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.079413] LNet: 28080:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.082852] LNet: 28080:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 464.098591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.150831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.374710] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:47:44 (1753440464) [ 469.584452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg228-server [ 470.854799] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:47:46 (1753440466) [ 470.993337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.048697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.087673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 471.108695] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 471.120569] LNet: Accept secure, port 988 [ 472.463711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[29-33]@tcp [ 472.478814] LNet: 29228:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 472.530463] LNetError: 29020:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.202.29@tcp is being used as a gateway but routing feature is not turned on [ 472.538723] LNetError: 29020:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.29@tcp has gone from up to down [ 476.845925] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:47:52 (1753440472) [ 476.905645] LNetError: 29815:0:(router.c:334:lnet_is_route_alive()) peer 192.168.202.33@tcp is being used as a gateway but routing feature is not turned on [ 477.884291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg228-server [ 478.843100] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:47:54 (1753440474) [ 478.960684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.039217] LNet: Removed LNI 192.168.202.28@tcp [ 480.059413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.103405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 480.116925] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 480.121672] LNet: Accept secure, port 988 [ 480.507167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[29-34/2]@tcp [ 480.520663] LNet: 30276:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 480.525498] LNet: 30276:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 480.542566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[29-34/2]@tcp [ 480.543765] LNetError: 30166:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.202.33@tcp is being used as a gateway but routing feature is not turned on [ 480.558335] LNetError: 30166:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 480.566282] LNetError: 30166:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.33@tcp has gone from up to down [ 480.573205] LNetError: 30166:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 484.453909] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:48:00 (1753440480) [ 484.546408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.606140] LNet: Removed LNI 192.168.202.28@tcp [ 485.625962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.710320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 485.753488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.787799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.827265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.872321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.900487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.931708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 485.981139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.003862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.054768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.103973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.125623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.156242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 486.208868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.230404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.285132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.326445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.348828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.389595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 486.434351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.456455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.509636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.565137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.598615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.628680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 486.667917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.687465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.722243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.759921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.779263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.812627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 486.849686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.876802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.927343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.780789] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:48:06 (1753440486) [ 492.072732] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg228-server [ 493.099454] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:48:08 (1753440488) [ 494.315668] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg228-server [ 495.655942] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:48:11 (1753440491) [ 496.177841] Key type .llcrypt unregistered [ 496.179142] Key type ._llcrypt unregistered [ 499.618992] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 499.719765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 499.722685] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 499.727245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 499.906572] Key type ._llcrypt registered [ 499.907975] Key type .llcrypt registered [ 500.049600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 500.058987] alg: No test for adler32 (adler32-zlib) [ 500.932601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.975028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 501.009250] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 501.013671] LNet: Accept secure, port 988 [ 501.483451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 501.514689] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 502.937973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 502.953571] LNet: Removed LNI 10.1.2.3@tcp [ 503.059864] LNet: 32955:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.073798] LNet: Removed LNI 192.168.202.28@tcp [ 503.593750] Key type .llcrypt unregistered [ 503.595372] Key type ._llcrypt unregistered [ 509.946535] Key type ._llcrypt registered [ 509.948118] Key type .llcrypt registered [ 510.634821] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:48:26 (1753440506) [ 510.961592] Key type .llcrypt unregistered [ 510.963289] Key type ._llcrypt unregistered [ 513.757798] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 513.821235] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 513.824898] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 513.828430] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 513.987528] Key type ._llcrypt registered [ 513.988925] Key type .llcrypt registered [ 514.111229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 514.120525] alg: No test for adler32 (adler32-zlib) [ 514.958770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.988610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 515.009219] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 515.016512] LNet: Accept secure, port 988 [ 515.447093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 515.465806] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 517.144961] LNet: 34411:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.153162] LNet: Removed LNI 192.168.202.28@tcp [ 517.160573] LNet: Removed LNI 10.1.2.3@tcp [ 517.583497] Key type .llcrypt unregistered [ 517.585249] Key type ._llcrypt unregistered [ 524.247357] Key type ._llcrypt registered [ 524.248865] Key type .llcrypt registered [ 525.223839] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:48:40 (1753440520) [ 525.538371] Key type .llcrypt unregistered [ 525.540173] Key type ._llcrypt unregistered [ 528.604103] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 528.713789] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 528.716551] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 528.718945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 528.890265] Key type ._llcrypt registered [ 528.892014] Key type .llcrypt registered [ 529.024571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 529.035960] alg: No test for adler32 (adler32-zlib) [ 529.883748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.929847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 529.955908] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 529.961174] LNet: Accept secure, port 988 [ 531.137974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 531.155205] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 532.352841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 532.368248] LNet: Removed LNI 10.1.2.3@tcp [ 532.377747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 532.402769] LNet: Removed LNI 10.1.2.31@tcp [ 532.413473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 532.436578] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 532.873562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 532.885429] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 533.388503] LNet: 36280:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.392037] LNet: Removed LNI 10.1.2.3@tcp [ 533.397882] LNet: Removed LNI 10.1.2.31@tcp [ 533.764929] Key type .llcrypt unregistered [ 533.766612] Key type ._llcrypt unregistered [ 539.214730] Key type ._llcrypt registered [ 539.216393] Key type .llcrypt registered [ 539.936201] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:48:55 (1753440535) [ 540.140836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 540.148912] alg: No test for adler32 (adler32-zlib) [ 540.984855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.017977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 541.031848] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 541.036269] LNet: Accept secure, port 988 [ 541.449360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.462918] LNet: 37231:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.479493] LNet: Removed LNI 192.168.202.28@tcp [ 541.497921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.530454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.28@tcp [ 541.549030] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 541.552849] LNet: Accept secure, port 988 [ 541.944241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.28@tcp [ 541.968548] LNet: Removed LNI 192.168.202.28@tcp [ 542.005621] LNet: 37363:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.354426] Key type .llcrypt unregistered [ 542.355820] Key type ._llcrypt unregistered [ 548.220633] Key type ._llcrypt registered [ 548.222475] Key type .llcrypt registered [ 548.933408] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:49:04 (1753440544) [ 549.952643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg228-server [ 550.813336] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:49:06 (1753440546) [ 551.019319] Key type .llcrypt unregistered [ 551.020751] Key type ._llcrypt unregistered [ 553.797287] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 553.840785] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 553.843168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 553.845770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 554.024488] Key type ._llcrypt registered [ 554.026510] Key type .llcrypt registered [ 554.119336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 554.128719] alg: No test for adler32 (adler32-zlib) [ 554.994299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.050583] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 555.053787] LNet: Accept secure, port 988 [ 555.470156] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 556.799388] LNet: 39162:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.804934] LNet: Removed LNI 192.168.202.28@tcp1 [ 556.815961] LNet: Removed LNI 10.1.2.3@tcp2 [ 557.188811] Key type .llcrypt unregistered [ 557.195951] Key type ._llcrypt unregistered [ 563.744250] Key type ._llcrypt registered [ 563.749119] Key type .llcrypt registered [ 564.577317] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 06:49:20 (1753440560) [ 564.815994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 564.825796] alg: No test for adler32 (adler32-zlib) [ 565.689918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.728567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 565.747775] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 565.751933] LNet: Accept secure, port 988 [ 566.279673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.28 [ 566.301229] LNet: Removed LNI 192.168.202.28@tcp [ 569.997072] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 06:49:25 (1753440565) [ 570.046987] LNet: 40680:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.058533] LNet: Removed LNI 192.168.202.28@tcp [ 570.583295] Key type .llcrypt unregistered [ 570.585057] Key type ._llcrypt unregistered [ 574.211227] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 574.339333] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 574.341708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 574.345337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 574.566272] Key type ._llcrypt registered [ 574.569972] Key type .llcrypt registered [ 574.659752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 574.669428] alg: No test for adler32 (adler32-zlib) [ 575.548727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 575.552952] LNet: Accept secure, port 988 [ 579.267314] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 06:49:35 (1753440575) [ 579.362974] LNet: 41551:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.377218] LNet: Removed LNI 10.1.2.3@tcp [ 579.746486] Key type .llcrypt unregistered [ 579.748131] Key type ._llcrypt unregistered [ 582.710021] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 582.907190] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 582.911279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 582.915571] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 583.113257] Key type ._llcrypt registered [ 583.114758] Key type .llcrypt registered [ 583.230134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 583.245899] alg: No test for adler32 (adler32-zlib) [ 584.108086] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 584.112619] LNet: Accept secure, port 988 [ 588.066341] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 06:49:43 (1753440583) [ 588.135901] LNet: 42412:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.153653] LNet: Removed LNI 10.1.2.3@tcp [ 588.554465] Key type .llcrypt unregistered [ 588.556159] Key type ._llcrypt unregistered [ 591.586260] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 591.731910] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 591.735425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 591.738887] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 591.915066] Key type ._llcrypt registered [ 591.916220] Key type .llcrypt registered [ 591.996752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 592.006739] alg: No test for adler32 (adler32-zlib) [ 592.868682] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 592.871373] LNet: Accept secure, port 988 [ 596.321665] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 06:49:52 (1753440592) [ 596.400861] LNet: 43276:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.430829] LNet: Removed LNI 10.1.2.3@tcp [ 596.911784] Key type .llcrypt unregistered [ 596.915631] Key type ._llcrypt unregistered [ 600.118193] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 600.274786] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 600.277328] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 600.282265] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 600.524382] Key type ._llcrypt registered [ 600.525982] Key type .llcrypt registered [ 600.670734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 600.681906] alg: No test for adler32 (adler32-zlib) [ 601.543850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.594116] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 601.596596] LNet: Accept secure, port 988 [ 606.657302] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 06:50:02 (1753440602) [ 606.923983] LNet: 44482:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.936830] LNet: Removed LNI 10.1.2.3@tcp [ 607.289452] Key type .llcrypt unregistered [ 607.291195] Key type ._llcrypt unregistered [ 610.248770] Key type ._llcrypt registered [ 610.250099] Key type .llcrypt registered [ 610.382985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 610.392735] alg: No test for adler32 (adler32-zlib) [ 611.274887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 611.302451] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 611.308249] LNet: Accept secure, port 988 [ 615.923881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 616.300773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_interrupt [ 616.327390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_interrupt [ 616.354019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 616.365707] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 616.513937] LNet: 44895:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 616.563403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 616.588534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 616.614107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 616.892431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_dropped [ 616.910526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_dropped [ 616.937961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 616.966047] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 617.105964] LNet: 44980:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 617.148311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 617.174273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 617.196234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 617.460475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_aborted [ 617.483464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_aborted [ 617.502603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 617.664164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 617.680728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 617.699521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 617.956105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_no_route [ 617.976941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_no_route [ 618.004489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 618.038233] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 618.044707] LNet: Skipped 1 previous similar message [ 618.171727] LNet: 45150:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 618.174307] LNet: 45150:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 618.209521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 618.226696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 618.247436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 618.513570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_timeout [ 618.537530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_timeout [ 618.555262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 618.723381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 618.744857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 618.771186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 618.991310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_error [ 619.010106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_error [ 619.029468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 619.188233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 619.212141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 619.231312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 619.358074] LNet: 45356:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.393468] LNet: Removed LNI 192.168.202.28@tcp [ 620.611389] Key type .llcrypt unregistered [ 620.613451] Key type ._llcrypt unregistered [ 627.812606] Key type ._llcrypt registered [ 627.816541] Key type .llcrypt registered [ 628.529521] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 06:50:24 (1753440624) [ 628.990416] Key type .llcrypt unregistered [ 628.992348] Key type ._llcrypt unregistered [ 631.445057] Key type ._llcrypt registered [ 631.446750] Key type .llcrypt registered [ 631.532901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 631.541130] alg: No test for adler32 (adler32-zlib) [ 632.360235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 632.375504] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 632.378356] LNet: Accept secure, port 988 [ 635.948789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 639.913818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.934121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 639.945334] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 640.583757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_interrupt [ 640.602733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -r 1 -e local_interrupt [ 640.621559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_interrupt [ 640.640264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -r 1 -e local_interrupt [ 640.659642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -r 1 -e local_interrupt [ 640.679509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -r 1 -e local_interrupt [ 640.697254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -r 1 -e local_interrupt [ 640.719544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -r 1 -e local_interrupt [ 640.741483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 640.753649] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 640.757503] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 640.880970] LNet: 46789:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 640.919200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 640.936707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 640.956615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 641.161334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_dropped [ 641.180169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -r 1 -e local_dropped [ 641.200744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_dropped [ 641.227642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -r 1 -e local_dropped [ 641.248356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -r 1 -e local_dropped [ 641.276406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -r 1 -e local_dropped [ 641.296756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -r 1 -e local_dropped [ 641.317370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -r 1 -e local_dropped [ 641.337775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 641.360459] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 641.368814] LNet: Skipped 2 previous similar messages [ 641.484893] LNet: 46886:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 641.529665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 641.549283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 641.567132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 641.821925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_aborted [ 641.840777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -r 1 -e local_aborted [ 641.860508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_aborted [ 641.880551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -r 1 -e local_aborted [ 641.903384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -r 1 -e local_aborted [ 641.924439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -r 1 -e local_aborted [ 641.945178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -r 1 -e local_aborted [ 641.965955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -r 1 -e local_aborted [ 641.988610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 642.164796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 642.182655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 642.200480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 642.417731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_no_route [ 642.437993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -r 1 -e local_no_route [ 642.466407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_no_route [ 642.486652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -r 1 -e local_no_route [ 642.506362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -r 1 -e local_no_route [ 642.528104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -r 1 -e local_no_route [ 642.550510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -r 1 -e local_no_route [ 642.571438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -r 1 -e local_no_route [ 642.597252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 642.617509] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 642.628984] LNet: Skipped 7 previous similar messages [ 642.776786] LNet: 47080:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 642.780340] LNet: 47080:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 642.819299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 642.838523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 642.857101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 643.085256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_timeout [ 643.107624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -r 1 -e local_timeout [ 643.128982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_timeout [ 643.154643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -r 1 -e local_timeout [ 643.175679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -r 1 -e local_timeout [ 643.208262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -r 1 -e local_timeout [ 643.231142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -r 1 -e local_timeout [ 643.252040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -r 1 -e local_timeout [ 643.275448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 643.514552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 643.538878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 643.557253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 643.803430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e local_error [ 643.830868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -r 1 -e local_error [ 643.850740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e local_error [ 643.872572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -r 1 -e local_error [ 643.893570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -r 1 -e local_error [ 643.914760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -r 1 -e local_error [ 643.935964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -r 1 -e local_error [ 643.956317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -r 1 -e local_error [ 643.977678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 644.149945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 644.176535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 644.199958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 646.722290] LNet: 47345:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.775459] LNet: Removed LNI 192.168.202.28@tcp [ 648.808823] LNet: Removed LNI 192.168.202.28@tcp1 [ 648.985660] Key type .llcrypt unregistered [ 648.987365] Key type ._llcrypt unregistered [ 656.079927] Key type ._llcrypt registered [ 656.083461] Key type .llcrypt registered [ 656.751809] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 06:50:52 (1753440652) [ 657.153720] Key type .llcrypt unregistered [ 657.156372] Key type ._llcrypt unregistered [ 659.825111] Key type ._llcrypt registered [ 659.826970] Key type .llcrypt registered [ 659.940737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 659.949232] alg: No test for adler32 (adler32-zlib) [ 660.775823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 660.796472] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 660.799465] LNet: Accept secure, port 988 [ 664.361338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 664.657561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e remote_dropped [ 664.677591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e remote_dropped [ 664.695742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 664.707219] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 664.826309] LNet: 48572:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 664.860543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 664.876433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 664.894219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 665.126145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e remote_error [ 665.144973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e remote_error [ 665.171395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 665.351662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 665.371671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 665.397337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 665.644093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e remote_timeout [ 665.663835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e remote_timeout [ 665.683512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 665.692855] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 665.696246] LNet: Skipped 1 previous similar message [ 665.840448] LNet: 48742:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 665.843789] LNet: 48742:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 665.884172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 665.902704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 665.924720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 666.045902] LNet: 48778:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.054883] LNet: Removed LNI 192.168.202.28@tcp [ 666.278736] Key type .llcrypt unregistered [ 666.280171] Key type ._llcrypt unregistered [ 672.973769] Key type ._llcrypt registered [ 672.975271] Key type .llcrypt registered [ 673.702734] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 06:51:09 (1753440669) [ 674.088539] Key type .llcrypt unregistered [ 674.090400] Key type ._llcrypt unregistered [ 676.618731] Key type ._llcrypt registered [ 676.620318] Key type .llcrypt registered [ 676.727420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 676.736511] alg: No test for adler32 (adler32-zlib) [ 677.562514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 677.578126] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 677.580817] LNet: Accept secure, port 988 [ 681.348819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 684.973808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.995607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 685.008719] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 685.670283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e remote_dropped [ 685.692804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -r 1 -e remote_dropped [ 685.716664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e remote_dropped [ 685.736201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -r 1 -e remote_dropped [ 685.759416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -r 1 -e remote_dropped [ 685.778713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -r 1 -e remote_dropped [ 685.803113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -r 1 -e remote_dropped [ 685.827626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -r 1 -e remote_dropped [ 685.850771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 685.861994] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 685.867975] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 685.989604] LNet: 50212:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 686.022051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 686.038549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 686.055278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 686.073864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 686.337292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e remote_error [ 686.363619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -r 1 -e remote_error [ 686.405047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e remote_error [ 686.435512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -r 1 -e remote_error [ 686.468889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -r 1 -e remote_error [ 686.493566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -r 1 -e remote_error [ 686.519098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -r 1 -e remote_error [ 686.546331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -r 1 -e remote_error [ 686.566402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 686.583295] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 686.587629] LNet: Skipped 2 previous similar messages [ 686.714239] LNet: 50311:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 686.756273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 686.774590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 686.796100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 686.815147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 687.078601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e remote_timeout [ 687.102526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -r 1 -e remote_timeout [ 687.123501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e remote_timeout [ 687.144949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -r 1 -e remote_timeout [ 687.166546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -r 1 -e remote_timeout [ 687.187706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -r 1 -e remote_timeout [ 687.216283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -r 1 -e remote_timeout [ 687.242869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -r 1 -e remote_timeout [ 687.264767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 687.446164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 687.468374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 687.491518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 687.540723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 690.238846] LNet: 50482:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.295157] LNet: Removed LNI 192.168.202.28@tcp [ 691.303605] LNet: Removed LNI 192.168.202.28@tcp1 [ 691.514768] Key type .llcrypt unregistered [ 691.516675] Key type ._llcrypt unregistered [ 698.900788] Key type ._llcrypt registered [ 698.902670] Key type .llcrypt registered [ 699.661902] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 06:51:35 (1753440695) [ 699.894503] Key type .llcrypt unregistered [ 699.895828] Key type ._llcrypt unregistered [ 702.398507] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 702.488072] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 702.493369] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 702.498703] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 702.673364] Key type ._llcrypt registered [ 702.674507] Key type .llcrypt registered [ 702.761452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 702.775468] alg: No test for adler32 (adler32-zlib) [ 703.604298] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 703.606923] LNet: Accept secure, port 988 [ 703.626650] LNet: 51547:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.637723] LNet: Removed LNI 192.168.202.28@tcp [ 703.956876] Key type .llcrypt unregistered [ 703.958569] Key type ._llcrypt unregistered [ 706.518725] Key type ._llcrypt registered [ 706.520327] Key type .llcrypt registered [ 706.617346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 706.625086] alg: No test for adler32 (adler32-zlib) [ 707.448087] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 707.450187] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 707.452629] LNet: Accept secure, port 988 [ 707.469307] LNet: 51816:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.472786] LNet: Removed LNI 192.168.202.28@tcp [ 707.482550] LNet: Removed LNI 10.1.2.3@tcp1 [ 707.794601] Key type .llcrypt unregistered [ 707.796706] Key type ._llcrypt unregistered [ 710.187344] Key type ._llcrypt registered [ 710.189039] Key type .llcrypt registered [ 710.281591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 710.291649] alg: No test for adler32 (adler32-zlib) [ 711.115699] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 711.117795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 711.121385] LNet: Accept secure, port 988 [ 711.151589] LNet: 52085:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.155410] LNet: Removed LNI 192.168.202.28@tcp [ 711.167499] LNet: Removed LNI 10.1.2.3@tcp [ 711.488901] Key type .llcrypt unregistered [ 711.490936] Key type ._llcrypt unregistered [ 714.007645] Key type ._llcrypt registered [ 714.009245] Key type .llcrypt registered [ 714.125899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.134536] alg: No test for adler32 (adler32-zlib) [ 714.981690] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 714.984573] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 714.987853] LNet: Accept secure, port 988 [ 715.006886] LNet: 52354:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.011590] LNet: Removed LNI 192.168.202.28@tcp [ 715.019508] LNet: Removed LNI 10.1.2.3@tcp [ 715.341417] Key type .llcrypt unregistered [ 715.342696] Key type ._llcrypt unregistered [ 717.770205] Key type ._llcrypt registered [ 717.771608] Key type .llcrypt registered [ 717.858885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 717.869627] alg: No test for adler32 (adler32-zlib) [ 718.688659] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 718.690683] LNetError: 52614:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 718.694027] LNetError: Error -100 starting up LNI tcp [ 718.701692] LNet: Removed LNI 192.168.202.28@tcp [ 719.021672] Key type .llcrypt unregistered [ 719.023313] Key type ._llcrypt unregistered [ 724.495257] Key type ._llcrypt registered [ 724.496927] Key type .llcrypt registered [ 725.204964] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 06:52:01 (1753440721) [ 725.655538] Key type .llcrypt unregistered [ 725.657596] Key type ._llcrypt unregistered [ 728.168032] Key type ._llcrypt registered [ 728.169798] Key type .llcrypt registered [ 728.291582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 728.312819] alg: No test for adler32 (adler32-zlib) [ 729.137715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 729.155929] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 729.159813] LNet: Accept secure, port 988 [ 733.009457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 733.292204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e network_timeout [ 733.312499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e network_timeout [ 733.334480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 733.350671] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 733.480103] LNet: 53857:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 733.522637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 733.543778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 733.692303] LNet: 53891:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.701692] LNet: Removed LNI 192.168.202.28@tcp [ 733.944439] Key type .llcrypt unregistered [ 733.946116] Key type ._llcrypt unregistered [ 741.991952] Key type ._llcrypt registered [ 741.994553] Key type .llcrypt registered [ 742.105602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.115450] alg: No test for adler32 (adler32-zlib) [ 742.935911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 742.952478] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 742.955397] LNet: Accept secure, port 988 [ 746.313661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 749.787596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.806267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 749.820406] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 750.485553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -r 1 -e network_timeout [ 750.505726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -r 1 -e network_timeout [ 750.530987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -r 1 -e network_timeout [ 750.554046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -r 1 -e network_timeout [ 750.574309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -r 1 -e network_timeout [ 750.599952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -r 1 -e network_timeout [ 750.618484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -r 1 -e network_timeout [ 750.638319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -r 1 -e network_timeout [ 750.660525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 750.670994] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 750.803058] LNet: 54739:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 750.850333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 750.876418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 753.267996] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 753.431841] LNet: 54807:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.465651] LNet: Removed LNI 192.168.202.28@tcp [ 754.473879] LNet: Removed LNI 192.168.202.28@tcp1 [ 754.726550] Key type .llcrypt unregistered [ 754.729412] Key type ._llcrypt unregistered [ 762.546666] Key type ._llcrypt registered [ 762.548747] Key type .llcrypt registered [ 763.302906] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 06:52:39 (1753440759) [ 763.479248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 763.489527] alg: No test for adler32 (adler32-zlib) [ 764.351902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.404346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 764.423563] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 764.426177] LNet: Accept secure, port 988 [ 764.848612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 764.859940] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 765.318708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 765.367827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 765.402153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 765.422172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 765.446785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 765.465860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 765.485610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 765.499966] LNetError: 55912:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 766.432780] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 766.432824] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 768.481042] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 768.755485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 772.575937] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 773.133603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 773.155848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 773.186859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.199492] LNet: 56029:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.204449] LNet: Removed LNI 192.168.202.28@tcp [ 773.212316] LNet: Removed LNI 192.168.202.28@tcp1 [ 773.223787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.251083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.262780] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 773.265435] LNet: Accept secure, port 988 [ 773.658134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 773.679779] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 774.126514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 774.150164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 774.178488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 774.204866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 774.245101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 774.267367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 774.292793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 774.311877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 774.325258] LNetError: 56295:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 775.263375] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 777.578809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 781.407541] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 781.411678] LNet: Skipped 1 previous similar message [ 786.092703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 786.113157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 786.132138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 789.951404] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 06:53:05 (1753440785) [ 790.012815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.027734] LNet: 57016:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.071567] LNet: Removed LNI 192.168.202.28@tcp [ 791.082815] LNet: Removed LNI 192.168.202.28@tcp1 [ 791.117955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.164779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.185565] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 791.189437] LNet: Accept secure, port 988 [ 791.645264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 791.658918] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 792.118919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 792.159031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 792.185540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 792.213366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 792.239103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 792.266465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 792.295234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 0 [ 792.345976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 793.183321] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.28@tcp [ 793.183374] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 793.193556] LNet: Skipped 1 previous similar message [ 808.114082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 808.189304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 808.207550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 808.222356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 500 [ 808.283518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 815.642442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 815.676947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.687479] LNet: 57524:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.735483] LNet: Removed LNI 192.168.202.28@tcp [ 816.746855] LNet: Removed LNI 192.168.202.28@tcp1 [ 816.766715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.798119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 816.816414] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 816.819972] LNet: Accept secure, port 988 [ 817.183872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 817.194288] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 817.599430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 817.620953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 817.655069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 817.680145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 817.707333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 817.726388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 817.744760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 817.762077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 0 [ 818.856485] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 818.859438] LNet: Skipped 3 previous similar messages [ 819.987392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 828.465210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 828.483409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 828.499707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 831.527693] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 06:53:47 (1753440827) [ 831.725230] LNet: 58563:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.735203] LNet: Removed LNI 192.168.202.28@tcp [ 832.745187] LNet: Removed LNI 192.168.202.28@tcp1 [ 833.049136] Key type .llcrypt unregistered [ 833.051064] Key type ._llcrypt unregistered [ 836.104299] Key type ._llcrypt registered [ 836.106144] Key type .llcrypt registered [ 836.212348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 836.225800] alg: No test for adler32 (adler32-zlib) [ 837.079745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 837.096937] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 837.100493] LNet: Accept secure, port 988 [ 842.260083] Lustre: DEBUG MARKER: Initial discovery [ 842.315498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 843.637317] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 843.690189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 843.710036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 843.731345] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 843.743734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 843.768439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.128@tcp -r 1 -m GET -l 3 [ 843.789366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.128@tcp -r 1 -m GET -e local_error [ 843.811192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.128@tcp -r 1 -m PUT -l 6 [ 844.626373] Lustre: DEBUG MARKER: Force oleg228-client.virtnet to discover 192.168.202.128@tcp (in background) [ 844.686846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 845.553412] Lustre: DEBUG MARKER: Force oleg228-server to discover 192.168.202.28@tcp [ 846.947944] Lustre: DEBUG MARKER: Wait for 59315 [ 847.713700] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 848.415782] Lustre: DEBUG MARKER: Finished wait on 59315 [ 848.473634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 848.498812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 848.633513] LNet: 59767:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.644639] LNet: Removed LNI 192.168.202.28@tcp [ 848.867674] Key type .llcrypt unregistered [ 848.869714] Key type ._llcrypt unregistered [ 856.908085] Key type ._llcrypt registered [ 856.909727] Key type .llcrypt registered [ 857.674960] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 06:54:13 (1753440853) [ 857.961432] Key type .llcrypt unregistered [ 857.963310] Key type ._llcrypt unregistered [ 861.035097] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 861.124197] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 861.127059] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 861.130562] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 861.301090] Key type ._llcrypt registered [ 861.302601] Key type .llcrypt registered [ 861.421490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 861.428499] alg: No test for adler32 (adler32-zlib) [ 862.258447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.295567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 862.312808] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 862.317411] LNet: Accept secure, port 988 [ 862.783735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 862.794048] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 867.221619] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 06:54:23 (1753440863) [ 867.304187] LNet: 61807:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.308812] LNet: Removed LNI 192.168.202.28@tcp [ 867.324777] LNet: Removed LNI 10.1.2.3@tcp [ 867.649470] Key type .llcrypt unregistered [ 867.651241] Key type ._llcrypt unregistered [ 870.380942] Key type ._llcrypt registered [ 870.382601] Key type .llcrypt registered [ 870.484119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.495446] alg: No test for adler32 (adler32-zlib) [ 871.327930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.357554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.373785] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 871.376571] LNet: Accept secure, port 988 [ 871.718925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 871.731346] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 872.190256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.28@tcp [ 873.247171] LNetError: Unexpected error -22 connecting to 192.168.202.28@tcp at host 192.168.202.28:988 [ 875.125221] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 06:54:31 (1753440871) [ 875.203301] LNet: 62921:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.207548] LNet: Removed LNI 192.168.202.28@tcp [ 875.217540] LNet: Removed LNI 10.1.2.3@tcp [ 875.488430] Key type .llcrypt unregistered [ 875.490151] Key type ._llcrypt unregistered [ 877.826038] Key type ._llcrypt registered [ 877.827942] Key type .llcrypt registered [ 877.931097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 877.940075] alg: No test for adler32 (adler32-zlib) [ 878.772280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.800898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 878.817603] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 878.820939] LNet: Accept secure, port 988 [ 879.134730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 879.144961] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 879.561058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.28@tcp1 --nid 192.168.202.28@tcp2 [ 881.691194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp1 192.168.202.28@tcp2 [ 881.703568] LNetError: 63746:0:(lib-move.c:2225:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.28@tcp1 to 192.168.202.28@tcp2 - Route table may be misconfigured [ 881.716816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp2 192.168.202.28@tcp1 [ 885.020059] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 06:54:40 (1753440880) [ 885.091884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.106225] LNet: 64312:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.176647] LNet: Removed LNI 192.168.202.28@tcp1 [ 887.206396] LNet: Removed LNI 192.168.202.28@tcp2 [ 887.231839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.273845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 887.296227] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 887.298777] LNet: Accept secure, port 988 [ 887.741863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 887.757177] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 888.316800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 888.337924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.28@tcp [ 888.362841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp1 192.168.202.28@tcp1 [ 888.395893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.28@tcp -d 192.168.202.28@tcp -e network_timeout [ 888.421288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -e network_timeout [ 888.451765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -e network_timeout [ 888.472967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -e network_timeout [ 888.502457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 888.516048] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 888.527411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 888.597482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 888.665247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 889.311432] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 892.179939] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 06:54:48 (1753440888) [ 892.272692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.291338] LNet: 65171:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.343157] LNet: Removed LNI 192.168.202.28@tcp [ 894.374227] LNet: Removed LNI 192.168.202.28@tcp1 [ 894.392391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.452951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 894.641086] LNet: 65210:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.856500] Key type .llcrypt unregistered [ 894.858374] Key type ._llcrypt unregistered [ 899.760571] Key type ._llcrypt registered [ 899.762296] Key type .llcrypt registered [ 900.401674] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 06:54:56 (1753440896) [ 900.605609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 900.614503] alg: No test for adler32 (adler32-zlib) [ 901.450848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.187561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 903.052029] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 06:54:59 (1753440899) [ 903.103734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.117352] LNet: 66300:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.133551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.166740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.179359] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 903.181831] LNet: Accept secure, port 988 [ 903.501140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 903.512844] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 903.918270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 903.940242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp1 [ 903.962820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp1 [ 907.185885] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 06:55:03 (1753440903) [ 908.076994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg228-server [ 908.900926] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 06:55:05 (1753440905) [ 909.775802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg228-server [ 910.591562] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 06:55:06 (1753440906) [ 911.462689] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg228-server [ 912.319802] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 06:55:08 (1753440908) [ 913.278425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg228-server [ 914.019179] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 06:55:10 (1753440910) [ 914.815333] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg228-server [ 915.560465] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 06:55:11 (1753440911) [ 916.398984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg228-server [ 917.165992] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 06:55:13 (1753440913) [ 918.008528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg228-server [ 918.819545] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 06:55:14 (1753440914) [ 919.640771] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg228-server [ 920.565728] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 06:55:16 (1753440916) [ 920.614519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.627425] LNet: 69830:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.696710] LNet: Removed LNI 192.168.202.28@tcp [ 922.726155] LNet: Removed LNI 192.168.202.28@tcp1 [ 922.737349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.767917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.781510] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 922.784502] LNet: Accept secure, port 988 [ 923.137278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 923.180818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 923.267597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.279076] LNet: 69979:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.325425] LNet: Removed LNI 192.168.202.28@tcp [ 924.335169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.359245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 924.373216] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 924.376495] LNet: Accept secure, port 988 [ 924.761453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 924.861354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.925575] LNet: Removed LNI 192.168.202.28@tcp [ 925.944718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.980970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.996814] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 926.001618] LNet: Accept secure, port 988 [ 926.319934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 926.365658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 926.485552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.503295] LNet: 70265:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.506488] LNet: 70265:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 927.527271] LNet: Removed LNI 192.168.202.28@tcp [ 927.541438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.581608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 927.602793] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 927.607505] LNet: Accept secure, port 988 [ 928.017373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 928.146803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.192664] LNet: Removed LNI 192.168.202.28@tcp [ 929.212621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.246683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 929.260553] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 929.263706] LNet: Accept secure, port 988 [ 929.564624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 929.606907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 930.814618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.829498] LNet: 70561:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.835819] LNet: 70561:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 931.879866] LNet: Removed LNI 192.168.202.28@tcp [ 931.896442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.939838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 931.956520] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 931.960357] LNet: Accept secure, port 988 [ 932.351409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 932.473387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.542509] LNet: Removed LNI 192.168.202.28@tcp [ 933.554414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.584540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 933.605939] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 933.612651] LNet: Accept secure, port 988 [ 933.970153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 934.012511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 934.157477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.207545] LNet: Removed LNI 192.168.202.28@tcp [ 935.228831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.266552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 935.288916] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 935.293185] LNet: Accept secure, port 988 [ 935.697651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 936.900219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.959929] LNet: Removed LNI 192.168.202.28@tcp [ 937.981889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.027690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 938.043753] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 938.047598] LNet: Accept secure, port 988 [ 938.539902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 938.593499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 938.817727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.879537] LNet: Removed LNI 192.168.202.28@tcp [ 939.891974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.923845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 939.938910] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 939.943322] LNet: Accept secure, port 988 [ 940.349199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 941.604323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.615508] LNet: 71295:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.618282] LNet: 71295:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 942.696581] LNet: Removed LNI 192.168.202.28@tcp [ 942.713850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.747578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 942.771459] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 942.778912] LNet: Accept secure, port 988 [ 943.218195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 943.265522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 943.419588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.490450] LNet: Removed LNI 192.168.202.28@tcp [ 944.513908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.544719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 944.567901] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 944.570452] LNet: Accept secure, port 988 [ 944.983926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 946.291650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.365656] LNet: Removed LNI 192.168.202.28@tcp [ 947.379097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.405386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 947.418161] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 947.420124] LNet: Accept secure, port 988 [ 947.743483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 947.772855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.792191] LNet: Removed LNI 192.168.202.28@tcp [ 947.804699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.831686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 947.846186] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 947.848971] LNet: Accept secure, port 988 [ 948.178671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 951.206122] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 06:55:47 (1753440947) [ 951.262745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.279693] LNet: Removed LNI 192.168.202.28@tcp [ 951.290950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.322916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 951.337262] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 951.339913] LNet: Accept secure, port 988 [ 951.895765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.913724] LNet: Removed LNI 192.168.202.28@tcp231 [ 951.923881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.956383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 951.968797] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 951.971528] LNet: Accept secure, port 988 [ 951.973205] LNet: 72564:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 952.170859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 952.188811] LNet: Removed LNI 192.168.202.28@tcp231 [ 952.203320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 952.223969] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 952.778065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.805212] LNet: Removed LNI 192.168.202.28@tcp231 [ 952.814767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.848874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 952.863115] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/0] [ 952.865656] LNet: Accept secure, port 988 [ 952.867871] LNet: 72714:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 953.049928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 953.072842] LNet: Removed LNI 192.168.202.28@tcp231 [ 953.081498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 953.098655] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/0] [ 953.621477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.643298] LNet: Removed LNI 192.168.202.28@tcp231 [ 953.654234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.691801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 953.708155] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/60] [ 953.711471] LNet: Accept secure, port 988 [ 953.885730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 953.903113] LNet: Removed LNI 192.168.202.28@tcp231 [ 953.912934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 953.925493] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/60] [ 954.416833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.437659] LNet: Removed LNI 192.168.202.28@tcp231 [ 954.454078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.494079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 954.513668] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 954.516193] LNet: Accept secure, port 988 [ 954.518279] LNet: 73010:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 954.521985] LNet: 73010:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 954.725788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 954.745205] LNet: Removed LNI 192.168.202.28@tcp231 [ 954.759184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 954.773840] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 958.249379] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 06:55:54 (1753440954) [ 958.288251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.302843] LNet: 73711:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.306147] LNet: 73711:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 958.316131] LNet: Removed LNI 192.168.202.28@tcp231 [ 958.329501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.363761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 958.378315] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 958.381129] LNet: Accept secure, port 988 [ 958.709510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 958.734029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 958.758577] LNet: Removed LNI 192.168.202.28@tcp232 [ 958.769505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 958.787347] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 959.133334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 959.156325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 959.183353] LNet: Removed LNI 192.168.202.28@tcp232 [ 959.202636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 959.217507] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 959.536785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 959.555944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 959.583710] LNet: Removed LNI 192.168.202.28@tcp232 [ 959.592680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 959.606893] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 959.922913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 959.941867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 959.955626] LNet: Removed LNI 192.168.202.28@tcp232 [ 959.964757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 962.859986] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 06:55:58 (1753440958) [ 962.919591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.947723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.980745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.994477] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 962.996901] LNet: Accept secure, port 988 [ 963.325440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 963.355719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 963.381395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 963.416479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 963.436881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 963.457128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 966.315380] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 06:56:02 (1753440962) [ 966.352973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.371211] LNet: Removed LNI 192.168.202.28@tcp [ 966.382449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.415116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 966.429393] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 966.432154] LNet: Accept secure, port 988 [ 969.720614] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 06:56:05 (1753440965) [ 970.415948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 971.285131] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 06:56:07 (1753440967) [ 971.501557] LNet: Removed LNI 192.168.202.28@tcp [ 971.799379] Key type .llcrypt unregistered [ 971.801456] Key type ._llcrypt unregistered [ 974.391177] Key type ._llcrypt registered [ 974.392866] Key type .llcrypt registered [ 974.504746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.513582] alg: No test for adler32 (adler32-zlib) [ 975.339340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 975.353847] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 975.356691] LNet: Accept secure, port 988 [ 978.885507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 982.170707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.128@tcp [ 982.370625] LNet: 76953:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.385467] LNet: Removed LNI 192.168.202.28@tcp [ 982.643488] Key type .llcrypt unregistered [ 982.645269] Key type ._llcrypt unregistered [ 987.737797] Key type ._llcrypt registered [ 987.739195] Key type .llcrypt registered [ 988.354451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 989.167425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 989.878324] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 06:56:26 (1753440986) [ 990.036339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 990.044708] alg: No test for adler32 (adler32-zlib) [ 990.870787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.918527] LNet: 78036:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.188092] Key type .llcrypt unregistered [ 991.189864] Key type ._llcrypt unregistered [ 993.664946] Key type ._llcrypt registered [ 993.667538] Key type .llcrypt registered [ 993.769105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 993.778253] alg: No test for adler32 (adler32-zlib) [ 994.607337] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 994.609150] LNet: 78288:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 994.615056] LNet: Accept secure, port 988 [ 994.624537] LNetError: 78300:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.202.30@tcp is being used as a gateway but routing feature is not turned on [ 994.633460] LNetError: 78300:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) route to o2ib through 192.168.202.30@tcp has gone from up to down [ 994.724358] LNetError: 78319:0:(router.c:334:lnet_is_route_alive()) peer 192.168.202.29@tcp is being used as a gateway but routing feature is not turned on [ 994.849555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.860045] LNet: 78344:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.877789] LNet: Removed LNI 192.168.202.28@tcp [ 995.893301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 995.904843] LNet: 78346:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 998.575248] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 06:56:34 (1753440994) [ 999.323578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg228-server [ 1000.065627] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 06:56:36 (1753440996) [ 1000.776784] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1000.873637] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1000.875982] LNet: 79355:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1000.880601] LNet: 79355:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 1000.884209] LNet: Accept secure, port 988 [ 1000.893278] LNetError: 79370:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.202.29@tcp is being used as a gateway but routing feature is not turned on [ 1000.905437] LNetError: 79370:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 1000.910427] LNetError: 79370:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) route to o2ib through 192.168.202.29@tcp has gone from up to down [ 1000.917620] LNetError: 79370:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1002.503203] Key type lgssc registered [ 1002.996649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.528992] Key type lgssc unregistered [ 1014.781715] LNet: 80392:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.846991] LNet: Removed LNI 192.168.202.28@tcp [ 1016.638745] Key type .llcrypt unregistered [ 1016.640268] Key type ._llcrypt unregistered [ 1036.158625] Key type ._llcrypt registered [ 1036.164414] Key type .llcrypt registered [ 1037.880566] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:57:12 (1753441032) [ 1038.577422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1038.609824] alg: No test for adler32 (adler32-zlib) [ 1039.941529] LNetError: 81234:0:(api-ni.c:2619:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1040.550364] Key type .llcrypt unregistered [ 1040.551852] Key type ._llcrypt unregistered [ 1046.158743] Key type ._llcrypt registered [ 1046.160316] Key type .llcrypt registered [ 1047.602916] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:57:22 (1753441042) [ 1048.401279] Key type .llcrypt unregistered [ 1048.402884] Key type ._llcrypt unregistered [ 1055.304588] Key type ._llcrypt registered [ 1055.306336] Key type .llcrypt registered [ 1055.543726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1055.570789] alg: No test for adler32 (adler32-zlib) [ 1063.842357] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:57:39 (1753441059) [ 1064.265187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1064.294561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.377680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1064.414708] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1064.423124] LNet: Accept secure, port 988 [ 1065.362777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1065.409598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1065.458212] LNet: Removed LNI 192.168.202.28@tcp [ 1065.772151] LNet: 82815:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.365420] Key type .llcrypt unregistered [ 1066.367824] Key type ._llcrypt unregistered [ 1077.828038] Key type ._llcrypt registered [ 1077.829621] Key type .llcrypt registered [ 1078.965613] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:57:54 (1753441074) [ 1079.301335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1079.312455] alg: No test for adler32 (adler32-zlib) [ 1080.279355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.369239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1080.401800] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1080.408424] LNet: Accept secure, port 988 [ 1081.158668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1081.200298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.28@tcp [ 1081.222717] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1081.224967] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1081.230784] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1081, 0, 0 [ 1081.235469] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1081.238217] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1081.248315] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1081.252470] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1081, 0, 0 [ 1081.264475] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1081.266190] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1081.268209] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1081.269941] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1081, 0, 1131 [ 1081.276604] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1081.279258] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1081.282380] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1081.285217] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1081, 0, 1131 [ 1081.289362] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1086.204744] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:58:01 (1753441081) [ 1086.528706] LNet: 84356:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.589253] LNet: Removed LNI 192.168.202.28@tcp [ 1088.127508] Key type .llcrypt unregistered [ 1088.129277] Key type ._llcrypt unregistered [ 1092.152908] Key type ._llcrypt registered [ 1092.154108] Key type .llcrypt registered [ 1092.325285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.339301] alg: No test for adler32 (adler32-zlib) [ 1093.217302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1093.248862] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1093.256965] LNet: Accept secure, port 988 [ 1098.643587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1104.404150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.453814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1104.514836] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1105.641836] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1105.645587] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1105.722515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1105.739425] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1114.269650] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:58:30 (1753441110) [ 1114.536306] LNet: 85818:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.551261] LNet: Removed LNI 192.168.202.28@tcp [ 1116.575264] LNet: Removed LNI 192.168.202.28@tcp1 [ 1116.582796] LNet: Removed LNI 10.1.2.3@tcp99 [ 1117.105513] Key type .llcrypt unregistered [ 1117.108949] Key type ._llcrypt unregistered [ 1121.419712] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1121.421993] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1121.596700] Key type ._llcrypt registered [ 1121.599690] Key type .llcrypt registered [ 1121.737277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1121.752797] alg: No test for adler32 (adler32-zlib) [ 1122.665925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1122.726442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1122.763587] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1122.768479] LNet: Accept secure, port 988 [ 1123.354586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1123.365388] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1124.995930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1125.020691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1125.054989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1125.073477] LNetError: 86529:0:(peer.c:2185:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1125.096643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1125.135680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1125.440822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1125.493140] LNet: 86549:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.499838] LNet: Removed LNI 192.168.202.28@tcp [ 1125.510219] LNet: Removed LNI 10.1.2.3@tcp [ 1125.964381] Key type .llcrypt unregistered [ 1125.966227] Key type ._llcrypt unregistered [ 1133.518690] Key type ._llcrypt registered [ 1133.520355] Key type .llcrypt registered [ 1134.471951] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:58:50 (1753441130) [ 1134.739478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1134.746741] alg: No test for adler32 (adler32-zlib) [ 1135.628548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.675285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1135.698197] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1135.701709] LNet: Accept secure, port 988 [ 1136.293669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1136.387654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg228-client@tcp [ 1141.418252] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:58:57 (1753441137) [ 1141.720979] LNet: 88075:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.759601] LNet: Removed LNI 192.168.202.28@tcp [ 1143.240698] Key type .llcrypt unregistered [ 1143.243178] Key type ._llcrypt unregistered [ 1147.553119] Key type ._llcrypt registered [ 1147.554743] Key type .llcrypt registered [ 1147.689975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1147.716346] alg: No test for adler32 (adler32-zlib) [ 1148.608466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1148.629748] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1148.636089] LNet: Accept secure, port 988 [ 1148.656499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1153.882436] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:59:09 (1753441149) [ 1155.173221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1156.711465] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:59:12 (1753441152) [ 1157.963827] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1159.245563] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:59:14 (1753441154) [ 1159.383591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1159.408325] LNet: 89491:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.423695] LNet: Removed LNI 192.168.202.28@tcp [ 1159.440827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1159.492041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1159.514763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1159.936531] Key type .llcrypt unregistered [ 1159.938318] Key type ._llcrypt unregistered [ 1167.062352] Key type ._llcrypt registered [ 1167.063875] Key type .llcrypt registered [ 1167.847255] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:59:23 (1753441163) [ 1168.185248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1168.196360] alg: No test for adler32 (adler32-zlib) [ 1169.096715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1169.152798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1169.177518] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1169.181221] LNet: Accept secure, port 988 [ 1169.734790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1169.789763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1175.075988] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:59:30 (1753441170) [ 1175.240326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1175.261298] LNet: 91039:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1176.299244] LNet: Removed LNI 192.168.202.28@tcp [ 1176.323301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1176.392655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1176.414736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1181.181166] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1182.270762] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:59:38 (1753441178) [ 1182.439531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1182.491261] LNet: 91758:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.508675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1182.643455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1182.664098] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1182.666453] LNet: Accept secure, port 988 [ 1183.163112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1183.184118] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1183.777614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1183.804729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1183.847564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.28@tcp -m PUT -e local_timeout -r 1 [ 1183.889128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1183.941675] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1183.944094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1184.109049] LNetError: 91782:0:(lib-move.c:4909:lnet_parse()) 192.168.202.28@tcp, src 192.168.202.28@tcp: Dropping PUT (error -108 looking up sender) [ 1185.119692] LNet: 92058:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.143193] LNet: Removed LNI 192.168.202.28@tcp [ 1186.158689] LNet: Removed LNI 10.1.2.3@tcp [ 1186.406276] Key type .llcrypt unregistered [ 1186.407927] Key type ._llcrypt unregistered [ 1192.896907] Key type ._llcrypt registered [ 1192.898405] Key type .llcrypt registered [ 1193.615749] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:59:49 (1753441189) [ 1193.874774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1193.887696] alg: No test for adler32 (adler32-zlib) [ 1194.761326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1195.028259] LNet: 92907:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.429388] Key type .llcrypt unregistered [ 1195.431198] Key type ._llcrypt unregistered [ 1199.012198] Key type ._llcrypt registered [ 1199.013784] Key type .llcrypt registered [ 1199.154362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1199.163706] alg: No test for adler32 (adler32-zlib) [ 1200.042376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1200.082538] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1200.088305] LNet: Accept secure, port 988 [ 1205.422595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1205.583536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1206.054978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.128@tcp [ 1206.244246] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 1206.264678] LNet: 93332:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1206.316891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1206.344085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1206.737199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1206.915639] LNet: 93425:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1206.970492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1206.998532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.385732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.128@tcp [ 1207.672279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.710627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1208.061375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1208.229652] LNet: 93612:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1208.234502] LNet: 93612:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 1208.298923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1208.326669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1208.542288] LNet: 93653:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.575050] LNet: Removed LNI 192.168.202.28@tcp [ 1209.850809] Key type .llcrypt unregistered [ 1209.856408] Key type ._llcrypt unregistered [ 1218.604309] Key type ._llcrypt registered [ 1218.606074] Key type .llcrypt registered [ 1219.372378] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:00:15 (1753441215) [ 1219.576883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1219.585815] alg: No test for adler32 (adler32-zlib) [ 1220.476325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1220.718869] LNet: 94537:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.065810] Key type .llcrypt unregistered [ 1221.067494] Key type ._llcrypt unregistered [ 1224.373581] Key type ._llcrypt registered [ 1224.375300] Key type .llcrypt registered [ 1224.508883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1224.523846] alg: No test for adler32 (adler32-zlib) [ 1225.436811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1225.462764] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1225.467457] LNet: Accept secure, port 988 [ 1230.785676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1236.655904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1236.706270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1236.725863] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1237.462305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1237.817672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.128@tcp [ 1238.031246] LNet: 95161:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.091322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.113147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.597057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1238.772080] LNet: 95255:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.818292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.822624] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.128@tcp, 192.168.202.128@tcp1 [ 1238.836195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.282418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.128@tcp [ 1239.542846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.564940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.953534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1240.108933] LNet: 95441:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.113414] LNet: 95441:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 1240.173196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.206455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.521119] LNet: 95516:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1244.575669] LNet: Removed LNI 192.168.202.28@tcp [ 1244.584516] LNet: Removed LNI 192.168.202.28@tcp1 [ 1244.881878] Key type .llcrypt unregistered [ 1244.883750] Key type ._llcrypt unregistered [ 1255.511528] Key type ._llcrypt registered [ 1255.513039] Key type .llcrypt registered [ 1256.706513] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1016 sec ========== 07:00:52 (1753441252) [ 1257.466492] Key type .llcrypt unregistered [ 1257.468813] Key type ._llcrypt unregistered [ 1265.774441] Key type ._llcrypt registered [ 1265.776488] Key type .llcrypt registered [ 1266.096618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1266.107704] alg: No test for adler32 (adler32-zlib) [ 1267.089539] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1267.288034] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1267.291602] LNet: Accept secure, port 988 [ 1268.951191] Key type lgssc registered [ 1269.648492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1314.252712] Lustre: Mounted lustre-client [ 1318.261817] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1328.629890] LustreError: 98725:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff90d546179800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1328.649899] LustreError: 98725:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1328.680112] Lustre: Unmounted lustre-client [ 1347.453866] Key type lgssc unregistered [ 1347.652942] LNet: 99201:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.711226] LNet: Removed LNI 192.168.202.28@tcp [ 1349.183823] Key type .llcrypt unregistered [ 1349.185632] Key type ._llcrypt unregistered