[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 424324641 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002296] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006479] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009145] LSM: Security Framework initializing [ 0.010041] Yama: becoming mindful. [ 0.011053] SELinux: Initializing. [ 0.012109] *** VALIDATE selinux *** [ 0.020222] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024660] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027119] *** VALIDATE tmpfs *** [ 0.029435] *** VALIDATE proc *** [ 0.030178] *** VALIDATE cgroup *** [ 0.030857] *** VALIDATE cgroup2 *** [ 0.032107] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034016] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036024] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.040085] debug: unmapping init [mem 0xffffffffbcc59000-0xffffffffbcc60fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043465] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044014] ... version: 2 [ 0.045007] ... bit width: 48 [ 0.046007] ... generic registers: 4 [ 0.047008] ... value mask: 0000ffffffffffff [ 0.048009] ... max period: 00007fffffffffff [ 0.049008] ... fixed-purpose events: 3 [ 0.050008] ... event mask: 000000070000000f [ 0.052190] rcu: Hierarchical SRCU implementation. [ 0.054154] smp: Bringing up secondary CPUs ... [ 0.055811] x86: Booting SMP configuration: [ 0.056045] .... node #0, CPUs: #1 #2 #3 [ 0.059707] smp: Brought up 1 node, 4 CPUs [ 0.060758] smpboot: Max logical packages: 1 [ 0.061013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.228574] node 0 deferred pages initialised in 166ms [ 0.231161] devtmpfs: initialized [ 0.232193] x86/mm: Memory block size: 128MB [ 0.234751] gcov: version magic: 0x41383552 [ 0.236135] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.237052] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.238227] pinctrl core: initialized pinctrl subsystem [ 0.239134] [ 0.239581] ************************************************************* [ 0.240006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.241005] ** ** [ 0.242008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.243007] ** ** [ 0.244007] ** This means that this kernel is built to expose internal ** [ 0.245009] ** IOMMU data structures, which may compromise security on ** [ 0.246009] ** your system. ** [ 0.247010] ** ** [ 0.248014] ** If you see this message and you are not debugging the ** [ 0.249010] ** kernel, report this immediately to your vendor! ** [ 0.250010] ** ** [ 0.251013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252008] ************************************************************* [ 0.253723] NET: Registered protocol family 16 [ 0.254352] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.255035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.256037] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.257392] cpuidle: using governor menu [ 0.258946] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.261427] PCI: Using configuration type 1 for base access [ 0.263102] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.271094] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.272035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.274075] cryptd: max_cpu_qlen set to 1000 [ 0.277266] ACPI: Added _OSI(Module Device) [ 0.278012] ACPI: Added _OSI(Processor Device) [ 0.281027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.283012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.288000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.294580] ACPI: Interpreter enabled [ 0.295197] ACPI: PM: (supports S0 S3 S4 S5) [ 0.298015] ACPI: Using IOAPIC for interrupt routing [ 0.300168] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.305682] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.319223] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.322055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.325012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.328080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.332660] acpiphp: Slot [2] registered [ 0.334140] acpiphp: Slot [3] registered [ 0.336080] acpiphp: Slot [4] registered [ 0.337079] acpiphp: Slot [5] registered [ 0.338179] acpiphp: Slot [6] registered [ 0.340125] acpiphp: Slot [7] registered [ 0.342150] acpiphp: Slot [8] registered [ 0.343071] acpiphp: Slot [9] registered [ 0.344061] acpiphp: Slot [10] registered [ 0.346070] acpiphp: Slot [11] registered [ 0.347105] acpiphp: Slot [12] registered [ 0.349065] acpiphp: Slot [13] registered [ 0.350059] acpiphp: Slot [14] registered [ 0.351087] acpiphp: Slot [15] registered [ 0.353122] acpiphp: Slot [16] registered [ 0.354068] acpiphp: Slot [17] registered [ 0.356068] acpiphp: Slot [18] registered [ 0.357111] acpiphp: Slot [19] registered [ 0.358061] acpiphp: Slot [20] registered [ 0.359059] acpiphp: Slot [21] registered [ 0.361078] acpiphp: Slot [22] registered [ 0.362059] acpiphp: Slot [23] registered [ 0.363059] acpiphp: Slot [24] registered [ 0.364080] acpiphp: Slot [25] registered [ 0.365084] acpiphp: Slot [26] registered [ 0.367053] acpiphp: Slot [27] registered [ 0.368058] acpiphp: Slot [28] registered [ 0.369058] acpiphp: Slot [29] registered [ 0.370055] acpiphp: Slot [30] registered [ 0.372090] acpiphp: Slot [31] registered [ 0.373049] PCI host bridge to bus 0000:00 [ 0.374013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.376015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.379020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.381010] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.383016] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.385036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.388373] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.395000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.400707] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.409025] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.413083] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.417032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.418018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.420012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.422488] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.425667] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.427025] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.429650] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.434013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.442979] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.446881] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.450648] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.454015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.459010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.472018] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.480560] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.488017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.504026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.525022] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.542000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.544364] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.545538] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.547453] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.549224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.554066] iommu: Default domain type: Passthrough [ 0.557913] SCSI subsystem initialized [ 0.560149] ACPI: bus type USB registered [ 0.561153] usbcore: registered new interface driver usbfs [ 0.564118] usbcore: registered new interface driver hub [ 0.566103] usbcore: registered new device driver usb [ 0.568167] pps_core: LinuxPPS API ver. 1 registered [ 0.570006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.573053] PTP clock support registered [ 0.574136] EDAC MC: Ver: 3.0.0 [ 0.576120] PCI: Using ACPI for IRQ routing [ 0.577631] NetLabel: Initializing [ 0.579009] NetLabel: domain hash size = 128 [ 0.581010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.582059] NetLabel: unlabeled traffic allowed by default [ 0.584326] vgaarb: loaded [ 0.586131] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.588010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.596000] clocksource: Switched to clocksource kvm-clock [ 0.701562] VFS: Disk quotas dquot_6.6.0 [ 0.703404] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.706116] *** VALIDATE ramfs *** [ 0.707481] *** VALIDATE hugetlbfs *** [ 0.710455] pnp: PnP ACPI init [ 0.713228] pnp: PnP ACPI: found 6 devices [ 0.731701] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.734616] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.737071] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.738706] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.740376] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.742192] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.744275] NET: Registered protocol family 2 [ 0.746120] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.749856] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.752480] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.756692] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.759436] TCP: Hash tables configured (established 65536 bind 65536) [ 0.761661] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.763976] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.766259] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.768670] NET: Registered protocol family 1 [ 0.770583] RPC: Registered named UNIX socket transport module. [ 0.772281] RPC: Registered udp transport module. [ 0.773621] RPC: Registered tcp transport module. [ 0.774909] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.776673] NET: Registered protocol family 44 [ 0.777769] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.779208] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.780447] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.781990] PCI: CLS 0 bytes, default 64 [ 0.783373] Unpacking initramfs... [ 2.228546] debug: unmapping init [mem 0xffff99403cc64000-0xffff99403ffcffff] [ 2.231987] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.234077] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.236364] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.823116] Initialise system trusted keyrings [ 2.827968] Key type blacklist registered [ 2.831669] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.843716] zbud: loaded [ 2.851360] *** VALIDATE nfs *** [ 2.852688] *** VALIDATE nfs4 *** [ 2.855436] pstore: using deflate compression [ 2.861796] Platform Keyring initialized [ 3.032350] NET: Registered protocol family 38 [ 3.033803] Key type asymmetric registered [ 3.035543] Asymmetric key parser 'x509' registered [ 3.037425] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.040536] io scheduler mq-deadline registered [ 3.041983] io scheduler kyber registered [ 3.044516] io scheduler bfq registered [ 3.046325] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.048873] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.051352] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.054222] ACPI: Power Button [PWRF] [ 3.153371] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.249776] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.371578] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.409214] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.440368] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.447991] Non-volatile memory driver v1.3 [ 3.451403] Linux agpgart interface v0.103 [ 3.495020] virtio_blk virtio1: [vda] 132696 512-byte logical blocks (67.9 MB/64.8 MiB) [ 3.503550] vda: detected capacity change from 0 to 67940352 [ 3.525591] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.528391] vdb: detected capacity change from 0 to 1073741824 [ 3.538317] libphy: Fixed MDIO Bus: probed [ 3.543366] usbcore: registered new interface driver usbserial_generic [ 3.547141] usbserial: USB Serial support registered for generic [ 3.551402] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.563654] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.567185] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.574643] mousedev: PS/2 mouse device common for all mice [ 3.577435] rtc_cmos 00:05: RTC can wake from S4 [ 3.588234] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.592241] rtc_cmos 00:05: registered as rtc0 [ 3.597433] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.603980] intel_pstate: CPU model not supported [ 3.611803] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.613812] hid: raw HID events driver (C) Jiri Kosina [ 3.626353] usbcore: registered new interface driver usbhid [ 3.632482] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.635408] usbhid: USB HID core driver [ 3.635605] drop_monitor: Initializing network drop monitor service [ 3.635768] Initializing XFRM netlink socket [ 3.636168] NET: Registered protocol family 10 [ 3.639093] Segment Routing with IPv6 [ 3.653448] NET: Registered protocol family 17 [ 3.656630] mpls_gso: MPLS GSO support [ 3.663274] RAS: Correctable Errors collector initialized. [ 3.665460] AVX version of gcm_enc/dec engaged. [ 3.667488] AES CTR mode by8 optimization enabled [ 3.799142] sched_clock: Marking stable (3799126113, 0)->(4646843863, -847717750) [ 3.803815] registered taskstats version 1 [ 3.806578] Loading compiled-in X.509 certificates [ 3.809514] zswap: loaded using pool lzo/zbud [ 3.839148] Key type big_key registered [ 3.857487] Key type encrypted registered [ 3.858926] ima: No TPM chip found, activating TPM-bypass! [ 3.861325] ima: Allocated hash algorithm: sha1 [ 3.863195] ima: No architecture policies found [ 3.864925] evm: Initialising EVM extended attributes: [ 3.866487] evm: security.selinux [ 3.867595] evm: security.ima [ 3.868681] evm: security.capability [ 3.869789] evm: HMAC attrs: 0x1 [ 3.872780] rtc_cmos 00:05: setting system clock to 2025-07-24 13:40:32 UTC (1753364432) [ 3.880300] debug: unmapping init [mem 0xffffffffbdc03000-0xffffffffbddfffff] [ 3.883891] debug: unmapping init [mem 0xffffffffbc982000-0xffffffffbcc58fff] [ 3.900149] Write protecting the kernel read-only data: 28672k [ 3.911013] debug: unmapping init [mem 0xffffffffbb003000-0xffffffffbb1fffff] [ 3.914141] debug: unmapping init [mem 0xffffffffbb914000-0xffffffffbb9fffff] [ 3.971317] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.992776] systemd[1]: Detected virtualization kvm. [ 3.994534] systemd[1]: Detected architecture x86-64. [ 3.996328] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.030949] systemd[1]: No hostname configured. [ 4.032527] systemd[1]: Set hostname to . [ 4.035283] random: systemd: uninitialized urandom read (16 bytes read) [ 4.037756] systemd[1]: Initializing machine ID from random generator. [ 4.095866] random: ln: uninitialized urandom read (6 bytes read) [ 4.280695] random: systemd: uninitialized urandom read (16 bytes read) [ 4.289985] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.297708] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.306579] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.338990] device-mapper: uevent: version 1.0.3 [ 5.340987] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.770884] virtio_net virtio0 ens2: renamed from eth0 [ 7.220791] scsi host0: ata_piix [ 7.262133] scsi host1: ata_piix [ 7.264129] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.266043] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.395864] random: crng init done [ 12.397226] random: 7 urandom warning(s) missed due to ratelimiting [ 13.959807] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.252607] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.649865] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.071829] SELinux: Disabled at runtime. [ 17.164481] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.175901] systemd[1]: Detected virtualization kvm. [ 17.179145] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.196965] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.200963] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.211555] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.216210] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.220251] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.231206] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.234858] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 18.385444] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 18.946767] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.620362] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.677764] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.881241] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.921214] EDAC sbridge: Ver: 1.1.2 [ 22.268921] Key type dns_resolver registered [ 22.680421] NFS: Registering the id_resolver key type [ 22.682112] Key type id_resolver registered [ 22.683793] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg356-client login: [ 52.292129] libcfs: loading out-of-tree module taints kernel. [ 52.301166] Key type ._llcrypt registered [ 52.301938] Key type .llcrypt registered [ 52.569394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 52.578831] alg: No test for adler32 (adler32-zlib) [ 53.513170] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 53.768180] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 53.770600] LNet: Accept secure, port 988 [ 55.375169] Key type lgssc registered [ 55.920543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.652393] Lustre: Mounted lustre-client [ 118.225684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 137.495798] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing check_logdir /tmp/testlogs/ [ 139.232099] Lustre: lustre-OST0000-osc-ffff99408778a800: disconnect after 23s idle [ 143.398694] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing yml_node [ 148.005872] Lustre: DEBUG MARKER: Client: 2.16.57.3 [ 150.553168] Lustre: DEBUG MARKER: MDS: 2.16.57.3 [ 152.707410] Lustre: DEBUG MARKER: OSS: 2.16.57.3 [ 153.892427] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jul 24 09:43:01 EDT 2025 [ 162.705991] LustreError: 7898:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff99408778a800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 162.712796] LustreError: 7898:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 162.730184] Lustre: Unmounted lustre-client [ 174.718683] Key type lgssc unregistered [ 174.928594] LNet: 8344:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 175.979283] LNet: Removed LNI 192.168.203.56@tcp [ 176.515363] Key type .llcrypt unregistered [ 176.516932] Key type ._llcrypt unregistered [ 189.665103] Key type ._llcrypt registered [ 189.666757] Key type .llcrypt registered [ 189.763271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 189.775200] alg: No test for adler32 (adler32-zlib) [ 190.604435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 190.638369] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 190.642305] LNet: Accept secure, port 988 [ 190.653572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 192.623126] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 192.861555] LNet: 9216:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 192.875226] LNet: Removed LNI 192.168.203.56@tcp [ 193.257620] Key type .llcrypt unregistered [ 193.259423] Key type ._llcrypt unregistered [ 195.885505] Key type ._llcrypt registered [ 195.888855] Key type .llcrypt registered [ 196.615704] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:43:44 (1753364624) [ 196.829171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 196.836528] alg: No test for adler32 (adler32-zlib) [ 197.678057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.722311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 197.735178] LNet: 9622:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 197.754306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.822322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 197.838704] LNet: 9629:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 201.531875] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:43:49 (1753364629) [ 201.580341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 201.592844] LNet: 10199:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 201.608416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 201.660817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 205.120911] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:43:52 (1753364632) [ 205.206049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.236766] LNet: 10788:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 205.251817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.302399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 208.856691] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:43:56 (1753364636) [ 208.901990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 208.916467] LNet: 11367:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 208.932709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 208.972477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 212.502949] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:44:00 (1753364640) [ 212.554052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 212.580365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 212.641027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 212.861231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 216.878180] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:44:04 (1753364644) [ 216.941746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.958481] LNet: 12544:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 216.961024] LNet: 12544:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 216.990535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.044215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 217.333519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 217.560488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 221.669934] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:44:09 (1753364649) [ 221.747915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 221.777427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 221.842871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 225.782194] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:44:13 (1753364653) [ 225.876346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 225.892388] LNet: 13721:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 225.895126] LNet: 13721:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 225.912283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 225.974161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 226.004440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 226.234817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 226.254972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 226.502468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 226.522927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 226.760089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 226.787506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 226.997588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 227.016850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 227.228350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 227.247460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 227.456932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 227.480383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 227.706185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 227.734160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 227.983624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 228.006109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 228.197687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 228.219788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 228.416127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 228.433771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 228.642356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 228.683632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 228.899884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 228.927730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 232.265136] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:44:20 (1753364660) [ 232.341572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.361722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.394895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 232.412263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 235.512059] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:44:23 (1753364663) [ 235.578105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.609432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.649706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 235.669901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 238.915392] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:44:26 (1753364666) [ 239.005981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.049580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.094502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 239.110929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 242.298469] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:44:30 (1753364670) [ 242.395737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 242.420672] LNet: 16164:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 242.423510] LNet: 16164:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 242.437861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.494403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 242.515689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 245.646281] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:44:33 (1753364673) [ 245.724944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 245.761322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 245.809412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 245.830463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 249.030346] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:44:36 (1753364676) [ 249.126980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.155091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.190492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 249.230611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 252.521731] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:44:40 (1753364680) [ 252.564693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.586913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.627380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 252.826772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 255.533583] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:44:43 (1753364683) [ 255.583831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.603525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.691701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 255.784559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 258.633281] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:44:46 (1753364686) [ 258.752989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.772613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.846556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 258.986195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 262.373500] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:44:50 (1753364690) [ 262.483468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.517539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.796182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 262.977361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 266.673616] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:44:54 (1753364694) [ 266.736861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.755085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.789577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 266.814470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 270.390081] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:44:58 (1753364698) [ 270.493329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.524148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.566964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 270.600723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 273.815094] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:45:01 (1753364701) [ 273.926285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.946170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.981230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 276.865141] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:45:04 (1753364704) [ 276.907726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.922730] LNet: 22178:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.926557] LNet: 22178:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 276.939391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.977656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 279.706301] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:45:07 (1753364707) [ 279.742907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.765834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.813684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 280.017361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 283.183542] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:45:11 (1753364711) [ 283.238165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.273952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.340569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 283.361083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 287.331240] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:45:15 (1753364715) [ 287.388783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.437189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.496803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 287.514830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 290.888583] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:45:18 (1753364718) [ 290.948761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.969892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.014073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 291.036768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 294.174301] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:45:22 (1753364722) [ 294.257918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.277393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.317373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 294.339760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 297.414866] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:45:25 (1753364725) [ 297.497396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.528617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.556909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 297.601945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 297.612996] LNetError: 25711:0:(peer.c:2185:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 297.634392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 297.651751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 300.847553] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:45:28 (1753364728) [ 300.910924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.937986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.990736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.B0Du520yJO [ 301.032196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.B0Du520yJO [ 301.070881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.B0Du520yJO [ 304.554243] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:45:32 (1753364732) [ 304.637958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.668272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.698396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 304.723547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 304.755114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 307.699993] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:45:35 (1753364735) [ 307.767796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.794808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.830807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 307.848860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 307.867280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 307.884332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 307.898551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 307.915850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 307.930823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 307.948737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 307.966691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 307.984972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 308.000885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 308.016947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 308.033857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 308.051323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 308.068725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 308.091938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 308.114538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 308.134667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 308.151997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 308.162827] LNetError: 27510:0:(router.c:632:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 310.864274] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:45:38 (1753364738) [ 310.925368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.950494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.986893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 313.907740] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:45:41 (1753364741) [ 314.705453] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg356-server [ 315.555704] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:45:43 (1753364743) [ 315.634361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.664674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.693277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 315.713477] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 315.716134] LNet: Accept secure, port 988 [ 316.577568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[57-61]@tcp [ 316.589913] LNet: 29221:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 316.623411] LNetError: 29014:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 316.629271] LNetError: 29014:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.59@tcp has gone from up to down [ 319.648760] LNetError: 29020:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.58@tcp has gone from up to down [ 319.657085] LNetError: 29020:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 319.825611] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:45:47 (1753364747) [ 319.890513] LNetError: 29807:0:(router.c:334:lnet_is_route_alive()) peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 320.812840] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg356-server [ 321.665126] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:45:49 (1753364749) [ 321.731335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.794428] LNet: Removed LNI 192.168.203.56@tcp [ 322.810779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.843711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 322.862765] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 322.865340] LNet: Accept secure, port 988 [ 323.177254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[57-62/2]@tcp [ 323.189846] LNet: 30270:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 323.195323] LNet: 30270:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 323.210053] LNetError: 30160:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.203.57@tcp is being used as a gateway but routing feature is not turned on [ 323.211799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[57-62/2]@tcp [ 323.218786] LNetError: 30160:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 323.218811] LNetError: 30160:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.57@tcp has gone from up to down [ 323.218816] LNetError: 30160:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 326.253727] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:45:54 (1753364754) [ 326.324346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.398110] LNet: Removed LNI 192.168.203.56@tcp [ 327.412631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.461490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 327.484791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.502329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.539501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 327.568933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.589797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.627741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 327.662190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.679485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.710778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 327.749866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.769955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.794616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 327.837659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.859180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.891437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 327.928105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.946690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.974850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 328.013467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.032166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.066372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 328.101588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.119492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.146463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 328.185607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.205326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.245942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 328.292657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.314957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.347140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 328.382986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.409778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.451715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 331.063348] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:45:59 (1753364759) [ 331.772484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg356-server [ 332.442059] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:46:00 (1753364760) [ 333.210422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg356-server [ 333.891497] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:46:01 (1753364761) [ 334.225147] Key type .llcrypt unregistered [ 334.226710] Key type ._llcrypt unregistered [ 336.281825] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 336.338512] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 336.342147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 336.347519] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 336.465494] Key type ._llcrypt registered [ 336.466906] Key type .llcrypt registered [ 336.548938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 336.556672] alg: No test for adler32 (adler32-zlib) [ 337.384408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.411583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 337.424416] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 337.428069] LNet: Accept secure, port 988 [ 337.690592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 337.700332] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 338.556687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 338.567862] LNet: Removed LNI 10.1.2.3@tcp [ 338.625303] LNet: 32948:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.637783] LNet: Removed LNI 192.168.203.56@tcp [ 338.940497] Key type .llcrypt unregistered [ 338.942209] Key type ._llcrypt unregistered [ 343.546679] Key type ._llcrypt registered [ 343.548334] Key type .llcrypt registered [ 344.180239] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:46:12 (1753364772) [ 344.399646] Key type .llcrypt unregistered [ 344.401454] Key type ._llcrypt unregistered [ 346.657764] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 346.710155] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 346.712088] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 346.715177] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 346.841740] Key type ._llcrypt registered [ 346.843961] Key type .llcrypt registered [ 346.927827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 346.936381] alg: No test for adler32 (adler32-zlib) [ 347.752721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.780677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 347.795441] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 347.798328] LNet: Accept secure, port 988 [ 348.088300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 348.099923] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 349.117760] LNet: 34404:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.120284] LNet: Removed LNI 192.168.203.56@tcp [ 349.127441] LNet: Removed LNI 10.1.2.3@tcp [ 349.379647] Key type .llcrypt unregistered [ 349.381495] Key type ._llcrypt unregistered [ 354.045120] Key type ._llcrypt registered [ 354.046709] Key type .llcrypt registered [ 354.682766] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:46:22 (1753364782) [ 354.972520] Key type .llcrypt unregistered [ 354.974293] Key type ._llcrypt unregistered [ 357.625862] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 357.691546] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 357.695420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 357.698517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 357.832870] Key type ._llcrypt registered [ 357.834207] Key type .llcrypt registered [ 357.932637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 357.942240] alg: No test for adler32 (adler32-zlib) [ 358.777127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.812203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 358.833371] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 358.836048] LNet: Accept secure, port 988 [ 359.705107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 359.716337] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 360.679558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 360.690945] LNet: Removed LNI 10.1.2.3@tcp [ 360.699087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 360.718562] LNet: Removed LNI 10.1.2.31@tcp [ 360.726243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 360.738956] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 361.050316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 361.060549] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 361.466596] LNet: 36279:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.469684] LNet: Removed LNI 10.1.2.3@tcp [ 361.476351] LNet: Removed LNI 10.1.2.31@tcp [ 361.749671] Key type .llcrypt unregistered [ 361.751623] Key type ._llcrypt unregistered [ 366.795624] Key type ._llcrypt registered [ 366.797246] Key type .llcrypt registered [ 367.444251] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:46:35 (1753364795) [ 367.627778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 367.638267] alg: No test for adler32 (adler32-zlib) [ 368.465227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.502512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 368.519807] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 368.522412] LNet: Accept secure, port 988 [ 368.886822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.898665] LNet: 37224:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.912856] LNet: Removed LNI 192.168.203.56@tcp [ 368.922270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.949406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.56@tcp [ 368.962231] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 368.965522] LNet: Accept secure, port 988 [ 369.322870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.56@tcp [ 369.346660] LNet: Removed LNI 192.168.203.56@tcp [ 369.703633] Key type .llcrypt unregistered [ 369.705724] Key type ._llcrypt unregistered [ 374.213934] Key type ._llcrypt registered [ 374.215605] Key type .llcrypt registered [ 374.819942] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:46:42 (1753364802) [ 375.682666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg356-server [ 376.482843] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:46:44 (1753364804) [ 376.730369] Key type .llcrypt unregistered [ 376.732314] Key type ._llcrypt unregistered [ 379.085048] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 379.132122] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 379.134814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 379.137451] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 379.271371] Key type ._llcrypt registered [ 379.273192] Key type .llcrypt registered [ 379.354040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 379.363250] alg: No test for adler32 (adler32-zlib) [ 380.179936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.214551] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 380.218134] LNet: Accept secure, port 988 [ 380.553360] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 381.532731] LNet: 39153:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.536548] LNet: Removed LNI 192.168.203.56@tcp1 [ 381.549406] LNet: Removed LNI 10.1.2.3@tcp2 [ 381.857650] Key type .llcrypt unregistered [ 381.859814] Key type ._llcrypt unregistered [ 386.366391] Key type ._llcrypt registered [ 386.367791] Key type .llcrypt registered [ 386.924185] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:46:54 (1753364814) [ 387.096777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 387.103956] alg: No test for adler32 (adler32-zlib) [ 387.929243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.961408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 387.984792] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 387.987540] LNet: Accept secure, port 988 [ 388.348750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.56 [ 388.359436] LNet: Removed LNI 192.168.203.56@tcp [ 391.108516] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:46:59 (1753364819) [ 391.159119] LNet: 40670:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.177683] LNet: Removed LNI 192.168.203.56@tcp [ 391.487149] Key type .llcrypt unregistered [ 391.488895] Key type ._llcrypt unregistered [ 394.081598] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 394.190192] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 394.192966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 394.195742] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 394.374474] Key type ._llcrypt registered [ 394.376052] Key type .llcrypt registered [ 394.457975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 394.465825] alg: No test for adler32 (adler32-zlib) [ 395.301281] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 395.305749] LNet: Accept secure, port 988 [ 398.014792] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:47:06 (1753364826) [ 398.059524] LNet: 41538:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.068693] LNet: Removed LNI 10.1.2.3@tcp [ 398.344363] Key type .llcrypt unregistered [ 398.346258] Key type ._llcrypt unregistered [ 400.645417] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 400.760499] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 400.762771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 400.764230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 400.934951] Key type ._llcrypt registered [ 400.936287] Key type .llcrypt registered [ 401.010468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 401.018185] alg: No test for adler32 (adler32-zlib) [ 401.854307] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 401.856594] LNet: Accept secure, port 988 [ 404.658466] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:47:12 (1753364832) [ 404.785222] LNet: 42407:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.797337] LNet: Removed LNI 10.1.2.3@tcp [ 405.124309] Key type .llcrypt unregistered [ 405.126055] Key type ._llcrypt unregistered [ 407.705849] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 407.833178] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 407.836078] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 407.839695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 407.997235] Key type ._llcrypt registered [ 408.000229] Key type .llcrypt registered [ 408.088812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 408.095827] alg: No test for adler32 (adler32-zlib) [ 408.952065] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 408.954286] LNet: Accept secure, port 988 [ 412.042900] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:47:19 (1753364839) [ 412.099928] LNet: 43267:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.113424] LNet: Removed LNI 10.1.2.3@tcp [ 412.443488] Key type .llcrypt unregistered [ 412.445228] Key type ._llcrypt unregistered [ 415.018140] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 415.123660] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 415.125310] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 415.127797] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 415.302700] Key type ._llcrypt registered [ 415.304329] Key type .llcrypt registered [ 415.390560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 415.397586] alg: No test for adler32 (adler32-zlib) [ 416.213673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.246694] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 416.248677] LNet: Accept secure, port 988 [ 420.251988] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:47:28 (1753364848) [ 420.452970] LNet: 44470:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.462109] LNet: Removed LNI 10.1.2.3@tcp [ 420.743805] Key type .llcrypt unregistered [ 420.745432] Key type ._llcrypt unregistered [ 423.138228] Key type ._llcrypt registered [ 423.139876] Key type .llcrypt registered [ 423.219567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 423.226497] alg: No test for adler32 (adler32-zlib) [ 424.039939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 424.053514] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 424.056521] LNet: Accept secure, port 988 [ 427.284239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 427.517175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_interrupt [ 427.537484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_interrupt [ 427.555083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 427.566740] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 427.675462] LNet: 44882:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 427.711484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 427.729600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 427.746601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 427.931599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_dropped [ 427.951529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_dropped [ 427.972672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 428.126873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 428.142571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 428.158790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 428.345432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_aborted [ 428.362860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_aborted [ 428.378773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 428.387459] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 428.390091] LNet: Skipped 1 previous similar message [ 428.500585] LNet: 45052:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 428.504321] LNet: 45052:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 428.536333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 428.552184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 428.569046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 428.755443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_no_route [ 428.771242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_no_route [ 428.790616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 428.935951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 428.953559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 428.970467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 429.159948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_timeout [ 429.177203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_timeout [ 429.197921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 429.346365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 429.361368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 429.377418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 429.559935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_error [ 429.579156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_error [ 429.598756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 429.614411] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 429.618073] LNet: Skipped 2 previous similar messages [ 429.720877] LNet: 45307:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 429.724843] LNet: 45307:0:(api-ni.c:337:recovery_interval_set()) Skipped 2 previous similar messages [ 429.756470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 429.772096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 429.789177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 429.890630] LNet: 45343:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.949475] LNet: Removed LNI 192.168.203.56@tcp [ 431.117544] Key type .llcrypt unregistered [ 431.119385] Key type ._llcrypt unregistered [ 437.080148] Key type ._llcrypt registered [ 437.081710] Key type .llcrypt registered [ 437.639133] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:47:45 (1753364865) [ 437.993579] Key type .llcrypt unregistered [ 437.995481] Key type ._llcrypt unregistered [ 440.186447] Key type ._llcrypt registered [ 440.187970] Key type .llcrypt registered [ 440.269426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 440.277058] alg: No test for adler32 (adler32-zlib) [ 441.103985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 441.120881] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 441.122836] LNet: Accept secure, port 988 [ 444.406638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 448.119355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.139169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 448.150448] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 448.696941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_interrupt [ 448.715306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -r 1 -e local_interrupt [ 448.734466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_interrupt [ 448.752480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -r 1 -e local_interrupt [ 448.772165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -r 1 -e local_interrupt [ 448.790880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -r 1 -e local_interrupt [ 448.809766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -r 1 -e local_interrupt [ 448.829984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -r 1 -e local_interrupt [ 448.848436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 448.858505] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 448.862137] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 448.969883] LNet: 46799:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 449.002172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 449.019095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 449.033840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 449.209525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_dropped [ 449.227929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -r 1 -e local_dropped [ 449.249594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_dropped [ 449.268135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -r 1 -e local_dropped [ 449.288159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -r 1 -e local_dropped [ 449.306436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -r 1 -e local_dropped [ 449.327591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -r 1 -e local_dropped [ 449.351667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -r 1 -e local_dropped [ 449.370038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 449.383551] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 449.388593] LNet: Skipped 2 previous similar messages [ 449.497608] LNet: 46896:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 449.533302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 449.553466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 449.572762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 449.770456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_aborted [ 449.790404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -r 1 -e local_aborted [ 449.809648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_aborted [ 449.831070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -r 1 -e local_aborted [ 449.848906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -r 1 -e local_aborted [ 449.867653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -r 1 -e local_aborted [ 449.887792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -r 1 -e local_aborted [ 449.906340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -r 1 -e local_aborted [ 449.924626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 450.108099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 450.125348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 450.139469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 450.332391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_no_route [ 450.352450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -r 1 -e local_no_route [ 450.369119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_no_route [ 450.387639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -r 1 -e local_no_route [ 450.407321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -r 1 -e local_no_route [ 450.426756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -r 1 -e local_no_route [ 450.446422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -r 1 -e local_no_route [ 450.466954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -r 1 -e local_no_route [ 450.484380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 450.501967] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 450.506884] LNet: Skipped 6 previous similar messages [ 450.619953] LNet: 47090:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 450.623578] LNet: 47090:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 450.656313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 450.674749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 450.694333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 450.886383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_timeout [ 450.904673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -r 1 -e local_timeout [ 450.923229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_timeout [ 450.946683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -r 1 -e local_timeout [ 450.970266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -r 1 -e local_timeout [ 450.992848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -r 1 -e local_timeout [ 451.013071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -r 1 -e local_timeout [ 451.033255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -r 1 -e local_timeout [ 451.051744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 451.224899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 451.242129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 451.259457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 451.451899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e local_error [ 451.471546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -r 1 -e local_error [ 451.490100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e local_error [ 451.513080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -r 1 -e local_error [ 451.534369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -r 1 -e local_error [ 451.556571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -r 1 -e local_error [ 451.577366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -r 1 -e local_error [ 451.598259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -r 1 -e local_error [ 451.616680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 451.772466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 451.787702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 451.805542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 454.166512] LNet: 47355:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.199444] LNet: Removed LNI 192.168.203.56@tcp [ 455.211938] LNet: Removed LNI 192.168.203.56@tcp1 [ 455.398836] Key type .llcrypt unregistered [ 455.400275] Key type ._llcrypt unregistered [ 461.826750] Key type ._llcrypt registered [ 461.833357] Key type .llcrypt registered [ 462.480584] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:48:10 (1753364890) [ 462.873358] Key type .llcrypt unregistered [ 462.875143] Key type ._llcrypt unregistered [ 465.202719] Key type ._llcrypt registered [ 465.204217] Key type .llcrypt registered [ 465.304274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 465.311529] alg: No test for adler32 (adler32-zlib) [ 466.135617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 466.152526] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 466.155979] LNet: Accept secure, port 988 [ 469.712879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 469.991869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e remote_dropped [ 470.011560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e remote_dropped [ 470.031737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 470.043196] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 470.177426] LNet: 48575:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 470.214841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 470.231835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 470.253422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 470.475839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e remote_error [ 470.495453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e remote_error [ 470.520896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 470.706899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 470.727226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 470.747555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 470.969957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e remote_timeout [ 470.990924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e remote_timeout [ 471.009795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 471.020990] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 471.023746] LNet: Skipped 1 previous similar message [ 471.129488] LNet: 48745:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 471.133614] LNet: 48745:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 471.169923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 471.187927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 471.203442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 471.311644] LNet: 48781:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.319682] LNet: Removed LNI 192.168.203.56@tcp [ 471.497752] Key type .llcrypt unregistered [ 471.499544] Key type ._llcrypt unregistered [ 477.689426] Key type ._llcrypt registered [ 477.690864] Key type .llcrypt registered [ 478.295596] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:48:26 (1753364906) [ 478.652279] Key type .llcrypt unregistered [ 478.654389] Key type ._llcrypt unregistered [ 480.783573] Key type ._llcrypt registered [ 480.785189] Key type .llcrypt registered [ 480.866873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 480.874188] alg: No test for adler32 (adler32-zlib) [ 481.695749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 481.710854] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 481.713304] LNet: Accept secure, port 988 [ 485.162518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 488.413787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.431356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 488.440989] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 488.961935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e remote_dropped [ 488.980605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -r 1 -e remote_dropped [ 488.998099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e remote_dropped [ 489.015820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -r 1 -e remote_dropped [ 489.033746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -r 1 -e remote_dropped [ 489.052334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -r 1 -e remote_dropped [ 489.071806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -r 1 -e remote_dropped [ 489.089594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -r 1 -e remote_dropped [ 489.107737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 489.119653] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 489.123787] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 489.241443] LNet: 50214:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 489.280100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 489.295760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 489.315421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 489.332155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 489.534681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e remote_error [ 489.551532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -r 1 -e remote_error [ 489.570267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e remote_error [ 489.590138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -r 1 -e remote_error [ 489.607847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -r 1 -e remote_error [ 489.627596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -r 1 -e remote_error [ 489.645990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -r 1 -e remote_error [ 489.665939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -r 1 -e remote_error [ 489.684038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 489.698407] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 489.703357] LNet: Skipped 2 previous similar messages [ 489.817477] LNet: 50314:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 489.849519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 489.865059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 489.882201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 489.899632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 490.087837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e remote_timeout [ 490.104977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -r 1 -e remote_timeout [ 490.121955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e remote_timeout [ 490.139262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -r 1 -e remote_timeout [ 490.156931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -r 1 -e remote_timeout [ 490.174856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -r 1 -e remote_timeout [ 490.193719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -r 1 -e remote_timeout [ 490.214640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -r 1 -e remote_timeout [ 490.233669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 490.382136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 490.398270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 490.414483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 490.431000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 492.836685] LNet: 50485:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.855214] LNet: Removed LNI 192.168.203.56@tcp [ 493.867304] LNet: Removed LNI 192.168.203.56@tcp1 [ 494.126538] Key type .llcrypt unregistered [ 494.128298] Key type ._llcrypt unregistered [ 500.972869] Key type ._llcrypt registered [ 500.974074] Key type .llcrypt registered [ 501.589783] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:48:49 (1753364929) [ 501.830464] Key type .llcrypt unregistered [ 501.832114] Key type ._llcrypt unregistered [ 503.976373] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 504.029776] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 504.032533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 504.034663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 504.164630] Key type ._llcrypt registered [ 504.166089] Key type .llcrypt registered [ 504.234496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 504.241749] alg: No test for adler32 (adler32-zlib) [ 505.063642] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 505.066216] LNet: Accept secure, port 988 [ 505.082971] LNet: 51546:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.097380] LNet: Removed LNI 192.168.203.56@tcp [ 505.375506] Key type .llcrypt unregistered [ 505.377239] Key type ._llcrypt unregistered [ 507.719556] Key type ._llcrypt registered [ 507.720823] Key type .llcrypt registered [ 507.807378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 507.814694] alg: No test for adler32 (adler32-zlib) [ 508.645622] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 508.647907] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 508.653842] LNet: Accept secure, port 988 [ 508.673326] LNet: 51814:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 508.678231] LNet: Removed LNI 192.168.203.56@tcp [ 508.684722] LNet: Removed LNI 10.1.2.3@tcp1 [ 508.981468] Key type .llcrypt unregistered [ 508.983327] Key type ._llcrypt unregistered [ 511.550554] Key type ._llcrypt registered [ 511.552175] Key type .llcrypt registered [ 511.657659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 511.676742] alg: No test for adler32 (adler32-zlib) [ 512.517263] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 512.519422] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 512.521553] LNet: Accept secure, port 988 [ 512.538057] LNet: 52083:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.542561] LNet: Removed LNI 192.168.203.56@tcp [ 512.552106] LNet: Removed LNI 10.1.2.3@tcp [ 512.875793] Key type .llcrypt unregistered [ 512.877145] Key type ._llcrypt unregistered [ 516.011483] Key type ._llcrypt registered [ 516.012925] Key type .llcrypt registered [ 516.181591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 516.204596] alg: No test for adler32 (adler32-zlib) [ 517.040649] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 517.042922] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 517.045469] LNet: Accept secure, port 988 [ 517.068130] LNet: 52353:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.072649] LNet: Removed LNI 192.168.203.56@tcp [ 517.083976] LNet: Removed LNI 10.1.2.3@tcp [ 517.632511] Key type .llcrypt unregistered [ 517.634368] Key type ._llcrypt unregistered [ 523.233697] Key type ._llcrypt registered [ 523.239545] Key type .llcrypt registered [ 523.434457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 523.443452] alg: No test for adler32 (adler32-zlib) [ 524.395333] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 524.397377] LNetError: 52614:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 524.405544] LNetError: Error -100 starting up LNI tcp [ 524.416676] LNet: Removed LNI 192.168.203.56@tcp [ 525.178348] Key type .llcrypt unregistered [ 525.183183] Key type ._llcrypt unregistered [ 535.075348] Key type ._llcrypt registered [ 535.077018] Key type .llcrypt registered [ 536.200617] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:49:23 (1753364963) [ 536.970498] Key type .llcrypt unregistered [ 536.973319] Key type ._llcrypt unregistered [ 542.074676] Key type ._llcrypt registered [ 542.076288] Key type .llcrypt registered [ 542.263454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 542.272648] alg: No test for adler32 (adler32-zlib) [ 543.150555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 543.178667] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 543.181893] LNet: Accept secure, port 988 [ 550.294539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 550.822362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e network_timeout [ 550.846358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e network_timeout [ 550.874891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 550.899198] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 551.091956] LNet: 53843:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 551.154452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 551.178753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 551.401829] LNet: 53877:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.418587] LNet: Removed LNI 192.168.203.56@tcp [ 551.788482] Key type .llcrypt unregistered [ 551.793755] Key type ._llcrypt unregistered [ 564.108653] Key type ._llcrypt registered [ 564.111361] Key type .llcrypt registered [ 564.229709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 564.236779] alg: No test for adler32 (adler32-zlib) [ 565.078471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 565.095930] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 565.098996] LNet: Accept secure, port 988 [ 569.705094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 575.021860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.049306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 575.065330] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 575.889434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -r 1 -e network_timeout [ 575.911988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -r 1 -e network_timeout [ 575.940672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -r 1 -e network_timeout [ 575.968276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -r 1 -e network_timeout [ 575.991599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -r 1 -e network_timeout [ 576.020748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -r 1 -e network_timeout [ 576.044208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -r 1 -e network_timeout [ 576.069252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -r 1 -e network_timeout [ 576.099608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 576.110756] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 576.249514] LNet: 54727:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 576.297055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 576.324565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 579.227727] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 579.382189] LNet: 54795:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 580.447268] LNet: Removed LNI 192.168.203.56@tcp [ 580.456675] LNet: Removed LNI 192.168.203.56@tcp1 [ 580.719512] Key type .llcrypt unregistered [ 580.721204] Key type ._llcrypt unregistered [ 590.664723] Key type ._llcrypt registered [ 590.666078] Key type .llcrypt registered [ 591.505223] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:50:19 (1753365019) [ 591.703581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 591.711858] alg: No test for adler32 (adler32-zlib) [ 592.567902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.610563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.630852] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 592.634174] LNet: Accept secure, port 988 [ 593.068042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 593.085301] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 593.626656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 593.676488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 593.711307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 593.728717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 593.753462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 593.770709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 593.793612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 593.820570] LNetError: 55885:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 594.657742] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 594.657777] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 596.707091] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 597.155902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 600.800113] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 601.510532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 601.533201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 601.565698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.580452] LNet: 56002:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.585333] LNet: Removed LNI 192.168.203.56@tcp [ 601.595571] LNet: Removed LNI 192.168.203.56@tcp1 [ 601.605648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.637831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 601.651424] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 601.653865] LNet: Accept secure, port 988 [ 602.001615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 602.013754] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 602.428851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 602.454646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 602.507906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 602.536599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 602.557222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 602.585699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 602.610639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 602.632604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 602.644238] LNetError: 56268:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 604.706162] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 604.710458] LNet: Skipped 1 previous similar message [ 604.843228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 608.799563] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 613.338577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 613.357823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 613.376373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 616.811029] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:50:44 (1753365044) [ 616.870480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.881802] LNet: 56982:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.887414] LNet: Removed LNI 192.168.203.56@tcp [ 617.897985] LNet: Removed LNI 192.168.203.56@tcp1 [ 617.915927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.953945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 617.985076] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 617.988455] LNet: Accept secure, port 988 [ 618.443192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 618.457412] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 618.937914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 618.974149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 618.994098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 619.012584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 619.035348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 619.061896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 619.085683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 0 [ 619.135446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 619.999294] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.56@tcp [ 621.024373] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 621.027711] LNet: Skipped 1 previous similar message [ 635.964203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 636.032255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 636.047912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 636.064112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 500 [ 636.119684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 643.475145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 643.521946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.537424] LNet: 57494:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.575158] LNet: Removed LNI 192.168.203.56@tcp [ 644.584652] LNet: Removed LNI 192.168.203.56@tcp1 [ 644.595218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.622509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.640394] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 644.643511] LNet: Accept secure, port 988 [ 645.080350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 645.092934] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 645.537818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 645.558522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 645.599953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 645.620670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 645.640902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 645.665971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 645.685829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 645.707459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 0 [ 646.687707] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 646.693329] LNet: Skipped 3 previous similar messages [ 646.891475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 657.590303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 657.612497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 657.633646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 660.937293] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:51:28 (1753365088) [ 661.130337] LNet: 58546:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.175189] LNet: Removed LNI 192.168.203.56@tcp [ 662.183809] LNet: Removed LNI 192.168.203.56@tcp1 [ 662.516942] Key type .llcrypt unregistered [ 662.518222] Key type ._llcrypt unregistered [ 665.091298] Key type ._llcrypt registered [ 665.092852] Key type .llcrypt registered [ 665.192484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 665.200334] alg: No test for adler32 (adler32-zlib) [ 666.021438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 666.037702] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 666.041602] LNet: Accept secure, port 988 [ 670.438392] Lustre: DEBUG MARKER: Initial discovery [ 670.465392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 671.647801] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 671.679868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 671.705886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 671.721345] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 671.734600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 671.758437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.156@tcp -r 1 -m GET -l 3 [ 671.777318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.156@tcp -r 1 -m GET -e local_error [ 671.800414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.156@tcp -r 1 -m PUT -l 6 [ 672.541603] Lustre: DEBUG MARKER: Force oleg356-client.virtnet to discover 192.168.203.156@tcp (in background) [ 672.567114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 673.247246] Lustre: DEBUG MARKER: Force oleg356-server to discover 192.168.203.56@tcp [ 674.540581] Lustre: DEBUG MARKER: Wait for 59295 [ 675.616109] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 676.518891] Lustre: DEBUG MARKER: Finished wait on 59295 [ 676.550601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 676.581970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 676.744835] LNet: 59752:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.799641] LNet: Removed LNI 192.168.203.56@tcp [ 678.030693] Key type .llcrypt unregistered [ 678.032415] Key type ._llcrypt unregistered [ 685.977904] Key type ._llcrypt registered [ 685.979166] Key type .llcrypt registered [ 686.596079] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:51:54 (1753365114) [ 686.817517] Key type .llcrypt unregistered [ 686.819163] Key type ._llcrypt unregistered [ 689.546640] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 689.585415] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 689.587933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 689.591352] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 689.740680] Key type ._llcrypt registered [ 689.742388] Key type .llcrypt registered [ 689.849906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 689.857648] alg: No test for adler32 (adler32-zlib) [ 690.682340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.712332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 690.726312] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 690.729108] LNet: Accept secure, port 988 [ 691.069467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 691.080172] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 695.139587] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:52:03 (1753365123) [ 695.217324] LNet: 61786:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.221778] LNet: Removed LNI 192.168.203.56@tcp [ 695.233385] LNet: Removed LNI 10.1.2.3@tcp [ 695.618828] Key type .llcrypt unregistered [ 695.621485] Key type ._llcrypt unregistered [ 698.798764] Key type ._llcrypt registered [ 698.800431] Key type .llcrypt registered [ 698.924087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 698.937192] alg: No test for adler32 (adler32-zlib) [ 699.779723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.823601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 699.840953] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 699.844308] LNet: Accept secure, port 988 [ 700.213798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 700.224529] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.766959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.56@tcp [ 701.855536] LNetError: Unexpected error -22 connecting to 192.168.203.56@tcp at host 192.168.203.56:988 [ 703.204252] hrtimer: interrupt took 2245731 ns [ 705.030735] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:52:12 (1753365132) [ 705.190746] LNet: 62906:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.207419] LNet: Removed LNI 192.168.203.56@tcp [ 706.224829] LNet: Removed LNI 10.1.2.3@tcp [ 706.611596] Key type .llcrypt unregistered [ 706.616226] Key type ._llcrypt unregistered [ 710.292216] Key type ._llcrypt registered [ 710.293414] Key type .llcrypt registered [ 710.390495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 710.400350] alg: No test for adler32 (adler32-zlib) [ 711.246925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.282156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 711.295924] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 711.300307] LNet: Accept secure, port 988 [ 711.719680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 711.733139] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 712.230712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.56@tcp1 --nid 192.168.203.56@tcp2 [ 714.644582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp1 192.168.203.56@tcp2 [ 714.659354] LNetError: 63731:0:(lib-move.c:2237:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.56@tcp1 to 192.168.203.56@tcp2 - Route table may be misconfigured [ 714.676613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp2 192.168.203.56@tcp1 [ 718.413228] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:52:26 (1753365146) [ 718.529338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.548500] LNet: 64303:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.583245] LNet: Removed LNI 192.168.203.56@tcp1 [ 720.614798] LNet: Removed LNI 192.168.203.56@tcp2 [ 720.626802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.657085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 720.670395] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 720.673157] LNet: Accept secure, port 988 [ 721.040968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 721.054375] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 721.482713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 721.508710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.56@tcp [ 721.534195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp1 192.168.203.56@tcp1 [ 721.557351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.56@tcp -d 192.168.203.56@tcp -e network_timeout [ 721.577155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -e network_timeout [ 721.597271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -e network_timeout [ 721.623969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -e network_timeout [ 721.649687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 721.664877] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 721.676963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 721.695234] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 721.737878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 721.792132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 724.927175] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:52:32 (1753365152) [ 725.041666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.061549] LNet: 65160:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.112425] LNet: Removed LNI 192.168.203.56@tcp [ 727.141994] LNet: Removed LNI 192.168.203.56@tcp1 [ 727.171061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.225535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 727.358772] LNet: 65196:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.583654] Key type .llcrypt unregistered [ 727.584860] Key type ._llcrypt unregistered [ 733.464092] Key type ._llcrypt registered [ 733.465628] Key type .llcrypt registered [ 734.239891] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:52:42 (1753365162) [ 734.495240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.508877] alg: No test for adler32 (adler32-zlib) [ 735.341452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.264932] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 737.591851] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:52:45 (1753365165) [ 737.681570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.696258] LNet: 66288:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.714762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.748419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.781580] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 737.785222] LNet: Accept secure, port 988 [ 738.227210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 738.242912] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 738.888786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 738.919378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp1 [ 738.965722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp1 [ 742.702048] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:52:50 (1753365170) [ 743.991471] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg356-server [ 745.180579] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:52:52 (1753365172) [ 746.403946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg356-server [ 747.525302] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:52:55 (1753365175) [ 748.743786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg356-server [ 749.834291] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:52:57 (1753365177) [ 751.027685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg356-server [ 752.191684] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:52:59 (1753365179) [ 753.269873] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg356-server [ 754.200166] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:53:02 (1753365182) [ 755.249483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg356-server [ 756.277592] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:53:04 (1753365184) [ 757.349676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg356-server [ 758.399657] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:53:06 (1753365186) [ 759.531483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg356-server [ 760.594702] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:53:08 (1753365188) [ 760.687935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.713220] LNet: 69817:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.759190] LNet: Removed LNI 192.168.203.56@tcp [ 762.790304] LNet: Removed LNI 192.168.203.56@tcp1 [ 762.808534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.849829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 762.878444] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 762.882946] LNet: Accept secure, port 988 [ 763.371935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 763.427573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 763.546418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.560639] LNet: 69964:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.581734] LNet: Removed LNI 192.168.203.56@tcp [ 764.599539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.631973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 764.654721] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 764.657620] LNet: Accept secure, port 988 [ 765.114449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 765.235980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.312934] LNet: Removed LNI 192.168.203.56@tcp [ 766.331715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.363521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 766.380810] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 766.383815] LNet: Accept secure, port 988 [ 766.762805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 766.832906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 768.043173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.060478] LNet: 70260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.063739] LNet: 70260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 769.126621] LNet: Removed LNI 192.168.203.56@tcp [ 769.137783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.168651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 769.185342] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 769.188072] LNet: Accept secure, port 988 [ 769.635302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 769.776584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.855106] LNet: Removed LNI 192.168.203.56@tcp [ 770.868887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.915334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 770.939904] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 770.945990] LNet: Accept secure, port 988 [ 771.357396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 771.402261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 773.644805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.655659] LNet: 70564:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.659219] LNet: 70564:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 774.694803] LNet: Removed LNI 192.168.203.56@tcp [ 774.714738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.751587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 774.771116] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 774.774709] LNet: Accept secure, port 988 [ 775.183907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 775.309757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.358966] LNet: Removed LNI 192.168.203.56@tcp [ 776.384070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.415331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.439314] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 776.444032] LNet: Accept secure, port 988 [ 776.877068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 776.917195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 777.062242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.151791] LNet: Removed LNI 192.168.203.56@tcp [ 778.171486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.221860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 778.271931] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 778.275024] LNet: Accept secure, port 988 [ 778.715500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 778.900292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.941355] LNet: Removed LNI 192.168.203.56@tcp [ 779.967918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.040699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.088666] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 780.092738] LNet: Accept secure, port 988 [ 780.504593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 780.555709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 780.736437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.797390] LNet: Removed LNI 192.168.203.56@tcp [ 781.808953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.848129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 781.868438] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 781.872302] LNet: Accept secure, port 988 [ 782.282658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 783.548331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.563841] LNet: 71289:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.568352] LNet: 71289:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 784.614867] LNet: Removed LNI 192.168.203.56@tcp [ 784.634602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.673081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.688624] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 784.691820] LNet: Accept secure, port 988 [ 785.087146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 785.134779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 787.502745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.586368] LNet: Removed LNI 192.168.203.56@tcp [ 788.606479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.660211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 788.686721] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 788.690961] LNet: Accept secure, port 988 [ 789.375188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 789.666810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.763317] LNet: Removed LNI 192.168.203.56@tcp [ 790.796786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.835768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 790.863277] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 790.866194] LNet: Accept secure, port 988 [ 791.313408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 791.355965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.379840] LNet: Removed LNI 192.168.203.56@tcp [ 791.393308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.427174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.442866] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 791.446635] LNet: Accept secure, port 988 [ 791.857380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 795.996297] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:53:43 (1753365223) [ 796.178585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.207488] LNet: Removed LNI 192.168.203.56@tcp [ 796.226435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.271779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 796.289277] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 796.293928] LNet: Accept secure, port 988 [ 796.978179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.993375] LNet: Removed LNI 192.168.203.56@tcp231 [ 797.010379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.048608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 797.065495] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 797.069860] LNet: Accept secure, port 988 [ 797.075463] LNet: 72569:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 797.290897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 797.316914] LNet: Removed LNI 192.168.203.56@tcp231 [ 797.334351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 797.357047] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 798.052731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.075590] LNet: Removed LNI 192.168.203.56@tcp231 [ 798.089393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.141156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 798.160510] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/0] [ 798.164694] LNet: Accept secure, port 988 [ 798.168489] LNet: 72718:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 798.416264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 798.449519] LNet: Removed LNI 192.168.203.56@tcp231 [ 798.463366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 798.493314] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/0] [ 799.110371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.136498] LNet: Removed LNI 192.168.203.56@tcp231 [ 799.147030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.194753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 799.212240] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/60] [ 799.217172] LNet: Accept secure, port 988 [ 799.219924] LNet: 72866:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 799.393101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 799.411863] LNet: Removed LNI 192.168.203.56@tcp231 [ 799.419990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 799.435256] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/60] [ 800.137593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.163500] LNet: 73006:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.167932] LNet: 73006:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 800.183890] LNet: Removed LNI 192.168.203.56@tcp231 [ 800.199659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.252620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 800.273242] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 800.279324] LNet: Accept secure, port 988 [ 800.520069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 800.542770] LNet: Removed LNI 192.168.203.56@tcp231 [ 800.554481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 800.572192] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 804.408669] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:53:52 (1753365232) [ 804.518228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.547143] LNet: Removed LNI 192.168.203.56@tcp231 [ 804.561447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.599448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 804.617945] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 804.623533] LNet: Accept secure, port 988 [ 805.003723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 805.027919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 805.046676] LNet: Removed LNI 192.168.203.56@tcp232 [ 805.056659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 805.075315] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 805.432572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 805.455183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 805.479760] LNet: Removed LNI 192.168.203.56@tcp232 [ 805.497705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 805.511336] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 805.927743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 805.951296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 805.978181] LNet: Removed LNI 192.168.203.56@tcp232 [ 805.989597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 806.005879] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 806.390752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 806.411279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 806.430915] LNet: Removed LNI 192.168.203.56@tcp232 [ 806.440840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 809.964527] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:53:57 (1753365237) [ 810.064772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.096892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.145426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.167599] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 810.170660] LNet: Accept secure, port 988 [ 810.635635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 810.695402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 810.726810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 810.764364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 810.792937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 810.818102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 814.075427] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:54:01 (1753365241) [ 814.183297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.209692] LNet: Removed LNI 192.168.203.56@tcp [ 814.222497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.252168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.266125] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 814.269154] LNet: Accept secure, port 988 [ 817.799267] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:54:05 (1753365245) [ 818.558345] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 819.341184] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:54:07 (1753365247) [ 819.565474] LNet: Removed LNI 192.168.203.56@tcp [ 819.828715] Key type .llcrypt unregistered [ 819.830352] Key type ._llcrypt unregistered [ 822.363271] Key type ._llcrypt registered [ 822.364814] Key type .llcrypt registered [ 822.480852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 822.490622] alg: No test for adler32 (adler32-zlib) [ 823.315845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 823.334885] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 823.337540] LNet: Accept secure, port 988 [ 827.332413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 830.795286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.156@tcp [ 830.961680] LNet: 76957:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.973447] LNet: Removed LNI 192.168.203.56@tcp [ 831.185394] Key type .llcrypt unregistered [ 831.187160] Key type ._llcrypt unregistered [ 836.265506] Key type ._llcrypt registered [ 836.267147] Key type .llcrypt registered [ 836.874000] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 837.644086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 838.377831] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:54:26 (1753365266) [ 838.543290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 838.550539] alg: No test for adler32 (adler32-zlib) [ 839.374188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.412623] LNet: 78040:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.682406] Key type .llcrypt unregistered [ 839.684092] Key type ._llcrypt unregistered [ 842.211695] Key type ._llcrypt registered [ 842.213404] Key type .llcrypt registered [ 842.310643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 842.318508] alg: No test for adler32 (adler32-zlib) [ 843.168411] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 843.170846] LNet: 78291:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 843.176917] LNet: Accept secure, port 988 [ 843.199533] LNetError: 78303:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.203.57@tcp is being used as a gateway but routing feature is not turned on [ 843.207925] LNetError: 78303:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.203.57@tcp has gone from up to down [ 843.303384] LNetError: 78323:0:(router.c:334:lnet_is_route_alive()) peer 192.168.203.57@tcp is being used as a gateway but routing feature is not turned on [ 843.442950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.919158] LNetError: 78303:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.203.60@tcp has gone from up to down [ 861.923612] LNetError: 78303:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 861.928507] LNet: 78348:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.949290] LNet: Removed LNI 192.168.203.56@tcp [ 862.965524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 862.976587] LNet: 78352:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 865.869840] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:54:53 (1753365293) [ 866.666160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg356-server [ 867.444752] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:54:55 (1753365295) [ 868.097756] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 868.203987] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 868.206019] LNet: 79361:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 868.210539] LNet: 79361:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 868.215121] LNet: Accept secure, port 988 [ 868.231648] LNetError: 79376:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 868.246391] LNetError: 79376:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 868.251066] LNetError: 79376:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.203.59@tcp has gone from up to down [ 868.255035] LNetError: 79376:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 869.823592] Key type lgssc registered [ 870.322796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 871.328261] LNetError: 79366:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.203.58@tcp has gone from up to down [ 871.335857] LNetError: 79366:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 879.939846] Key type lgssc unregistered [ 880.066411] LNet: 80398:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.126409] LNet: Removed LNI 192.168.203.56@tcp [ 881.425862] Key type .llcrypt unregistered [ 881.427451] Key type ._llcrypt unregistered [ 888.104887] Key type ._llcrypt registered [ 888.106727] Key type .llcrypt registered [ 888.726119] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:55:16 (1753365316) [ 888.858931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 888.865914] alg: No test for adler32 (adler32-zlib) [ 889.760110] LNetError: 81237:0:(api-ni.c:2619:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 889.966518] Key type .llcrypt unregistered [ 889.968062] Key type ._llcrypt unregistered [ 892.385334] Key type ._llcrypt registered [ 892.386818] Key type .llcrypt registered [ 892.934424] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:55:20 (1753365320) [ 893.241484] Key type .llcrypt unregistered [ 893.243524] Key type ._llcrypt unregistered [ 895.932969] Key type ._llcrypt registered [ 895.934502] Key type .llcrypt registered [ 896.026812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 896.039799] alg: No test for adler32 (adler32-zlib) [ 900.061608] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:55:28 (1753365328) [ 900.160698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.188136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.241656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 900.264784] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 900.267704] LNet: Accept secure, port 988 [ 900.705841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 900.730652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 900.753979] LNet: Removed LNI 192.168.203.56@tcp [ 900.856693] LNet: 82816:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.058427] Key type .llcrypt unregistered [ 901.060066] Key type ._llcrypt unregistered [ 905.766821] Key type ._llcrypt registered [ 905.769122] Key type .llcrypt registered [ 906.387545] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:55:34 (1753365334) [ 906.585153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 906.592243] alg: No test for adler32 (adler32-zlib) [ 907.424148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.452304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 907.468055] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 907.472539] LNet: Accept secure, port 988 [ 907.822759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 907.845985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.56@tcp [ 907.857645] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 907.860147] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 907.862327] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 907, 0, 0 [ 907.865610] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 907.868054] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 907.870068] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 907.872564] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 907, 0, 0 [ 907.875400] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 907.877110] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 907.879577] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 907.881977] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 907, 0, 957 [ 907.885185] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 907.887167] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 907.890026] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 907.891789] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 907, 0, 957 [ 907.895206] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 910.614097] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:55:38 (1753365338) [ 910.831448] LNet: 84355:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.846677] LNet: Removed LNI 192.168.203.56@tcp [ 912.148620] Key type .llcrypt unregistered [ 912.150418] Key type ._llcrypt unregistered [ 914.523384] Key type ._llcrypt registered [ 914.524816] Key type .llcrypt registered [ 914.612764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 914.620882] alg: No test for adler32 (adler32-zlib) [ 915.457314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 915.482919] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 915.486452] LNet: Accept secure, port 988 [ 919.454256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 923.104855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.126872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 923.137992] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 923.564196] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 923.566233] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 923.599207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 923.610867] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 928.198254] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:55:56 (1753365356) [ 928.295713] LNet: 85815:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.311144] LNet: Removed LNI 192.168.203.56@tcp [ 930.335155] LNet: Removed LNI 192.168.203.56@tcp1 [ 930.343662] LNet: Removed LNI 10.1.2.3@tcp99 [ 930.640638] Key type .llcrypt unregistered [ 930.642070] Key type ._llcrypt unregistered [ 933.034361] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 933.037813] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 933.165524] Key type ._llcrypt registered [ 933.167681] Key type .llcrypt registered [ 933.278547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 933.286640] alg: No test for adler32 (adler32-zlib) [ 934.110584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.137539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 934.151746] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 934.155115] LNet: Accept secure, port 988 [ 934.473440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 934.483407] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 935.375201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 935.400081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 935.422547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 935.433583] LNetError: 86526:0:(peer.c:2185:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 935.449227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 935.474464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 935.667990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 935.700776] LNet: 86546:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.735139] LNet: Removed LNI 192.168.203.56@tcp [ 936.741699] LNet: Removed LNI 10.1.2.3@tcp [ 937.014964] Key type .llcrypt unregistered [ 937.016585] Key type ._llcrypt unregistered [ 941.500090] Key type ._llcrypt registered [ 941.501843] Key type .llcrypt registered [ 942.088966] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:56:10 (1753365370) [ 942.235830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.245444] alg: No test for adler32 (adler32-zlib) [ 943.060973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.085345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.099418] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 943.102987] LNet: Accept secure, port 988 [ 943.426370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 943.465460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg356-client@tcp [ 946.345429] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:56:14 (1753365374) [ 946.390173] LNet: 88064:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.430128] LNet: Removed LNI 192.168.203.56@tcp [ 947.728819] Key type .llcrypt unregistered [ 947.730206] Key type ._llcrypt unregistered [ 950.153488] Key type ._llcrypt registered [ 950.156892] Key type .llcrypt registered [ 950.239929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 950.247357] alg: No test for adler32 (adler32-zlib) [ 951.054238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 951.072781] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 951.075854] LNet: Accept secure, port 988 [ 951.091522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 953.686573] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:56:21 (1753365381) [ 954.307955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 955.002213] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:56:23 (1753365383) [ 955.657382] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 956.388197] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:56:24 (1753365384) [ 956.450751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.463957] LNet: 89483:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.474636] LNet: Removed LNI 192.168.203.56@tcp [ 956.483518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.516943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 956.534692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 956.812415] Key type .llcrypt unregistered [ 956.814134] Key type ._llcrypt unregistered [ 961.416913] Key type ._llcrypt registered [ 961.418490] Key type .llcrypt registered [ 962.033866] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:56:29 (1753365389) [ 962.207919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 962.215422] alg: No test for adler32 (adler32-zlib) [ 963.033822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.061906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 963.077657] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 963.080030] LNet: Accept secure, port 988 [ 963.392939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 963.420090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 966.266330] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:56:34 (1753365394) [ 966.346422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.366693] LNet: 91030:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.397701] LNet: Removed LNI 192.168.203.56@tcp [ 967.407978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.435953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 967.452574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 969.962820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 970.658391] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:56:38 (1753365398) [ 970.696343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.706968] LNet: 91745:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.720947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.812919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 970.825535] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 970.828025] LNet: Accept secure, port 988 [ 971.162908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 971.174311] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 971.572322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 971.590838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 971.621308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.56@tcp -m PUT -e local_timeout -r 1 [ 971.653461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 971.687695] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 971.690840] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 971.801669] LNetError: 91773:0:(lib-move.c:4921:lnet_parse()) 192.168.203.56@tcp, src 192.168.203.56@tcp: Dropping PUT (error -108 looking up sender) [ 972.832111] LNet: 92049:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.855255] LNet: Removed LNI 192.168.203.56@tcp [ 973.863701] LNet: Removed LNI 10.1.2.3@tcp [ 974.032504] Key type .llcrypt unregistered [ 974.034157] Key type ._llcrypt unregistered [ 978.685737] Key type ._llcrypt registered [ 978.688089] Key type .llcrypt registered [ 979.378820] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:56:47 (1753365407) [ 979.530559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 979.537205] alg: No test for adler32 (adler32-zlib) [ 980.362168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.543218] LNet: 92899:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.833957] Key type .llcrypt unregistered [ 980.835660] Key type ._llcrypt unregistered [ 983.257492] Key type ._llcrypt registered [ 983.258893] Key type .llcrypt registered [ 983.354161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.362315] alg: No test for adler32 (adler32-zlib) [ 984.173912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 984.187767] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 984.190241] LNet: Accept secure, port 988 [ 987.801552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 987.892304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 988.129760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.156@tcp [ 988.253879] LNet: 93323:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 988.292332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.309121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 988.547798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 988.697221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.714920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 988.981348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.156@tcp [ 989.098696] LNet: 93509:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 989.102597] LNet: 93509:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 989.137805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 989.157444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 989.430455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 989.580079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 989.598488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 989.747634] LNet: 93643:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.822237] LNet: Removed LNI 192.168.203.56@tcp [ 991.035365] Key type .llcrypt unregistered [ 991.037190] Key type ._llcrypt unregistered [ 997.663445] Key type ._llcrypt registered [ 997.665085] Key type .llcrypt registered [ 998.261686] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:57:06 (1753365426) [ 998.465649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.473206] alg: No test for adler32 (adler32-zlib) [ 999.296728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.497816] LNet: 94528:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.783880] Key type .llcrypt unregistered [ 999.785597] Key type ._llcrypt unregistered [ 1002.285507] Key type ._llcrypt registered [ 1002.286746] Key type .llcrypt registered [ 1002.376935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.383914] alg: No test for adler32 (adler32-zlib) [ 1003.202798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1003.217447] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1003.220233] LNet: Accept secure, port 988 [ 1006.538517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1009.847689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.870072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1009.882163] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1010.298479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1010.530099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.156@tcp [ 1010.655942] LNet: 95151:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1010.689870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1010.708352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1010.942815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1011.083384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1011.100782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1011.332083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.156@tcp [ 1011.424071] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 1011.450064] LNet: 95338:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1011.453517] LNet: 95338:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 1011.492498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1011.509659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1011.750458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1011.894444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1011.911071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1014.285975] LNet: 95506:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.327130] LNet: Removed LNI 192.168.203.56@tcp [ 1015.335314] LNet: Removed LNI 192.168.203.56@tcp1 [ 1015.552894] Key type .llcrypt unregistered [ 1015.554422] Key type ._llcrypt unregistered [ 1021.989613] Key type ._llcrypt registered [ 1021.991094] Key type .llcrypt registered [ 1022.595174] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 868 sec =========== 09:57:30 (1753365450) [ 1022.868638] Key type .llcrypt unregistered [ 1022.870257] Key type ._llcrypt unregistered [ 1027.651186] Key type ._llcrypt registered [ 1027.654561] Key type .llcrypt registered [ 1027.831882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.840187] alg: No test for adler32 (adler32-zlib) [ 1028.716758] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 1028.816420] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1028.818988] LNet: Accept secure, port 988 [ 1030.409297] Key type lgssc registered [ 1030.886634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1057.307547] Lustre: Mounted lustre-client [ 1059.932861] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1067.776992] LustreError: 98695:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9940891f6800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1067.785949] LustreError: 98695:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1067.802298] Lustre: Unmounted lustre-client [ 1094.065296] Key type lgssc unregistered [ 1094.196550] LNet: 99173:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.269778] LNet: Removed LNI 192.168.203.56@tcp [ 1095.533133] Key type .llcrypt unregistered [ 1095.534790] Key type ._llcrypt unregistered