[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 385058855 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002312] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.007246] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008014] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009005] pid_max: default: 32768 minimum: 301 [ 0.010107] LSM: Security Framework initializing [ 0.011026] Yama: becoming mindful. [ 0.012020] SELinux: Initializing. [ 0.013038] *** VALIDATE selinux *** [ 0.020095] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024172] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026026] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029083] *** VALIDATE tmpfs *** [ 0.031399] *** VALIDATE proc *** [ 0.032137] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034178] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036098] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038018] Spectre V2 : User space: Vulnerable [ 0.039003] Speculative Store Bypass: Vulnerable [ 0.042453] debug: unmapping init [mem 0xffffffffad859000-0xffffffffad860fff] [ 0.044209] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045618] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046014] ... version: 2 [ 0.046785] ... bit width: 48 [ 0.047006] ... generic registers: 4 [ 0.047650] ... value mask: 0000ffffffffffff [ 0.048009] ... max period: 00007fffffffffff [ 0.049007] ... fixed-purpose events: 3 [ 0.049749] ... event mask: 000000070000000f [ 0.051179] rcu: Hierarchical SRCU implementation. [ 0.053362] smp: Bringing up secondary CPUs ... [ 0.054520] x86: Booting SMP configuration: [ 0.055019] .... node #0, CPUs: #1 #2 #3 [ 0.058378] smp: Brought up 1 node, 4 CPUs [ 0.060054] smpboot: Max logical packages: 1 [ 0.061030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.230307] node 0 deferred pages initialised in 168ms [ 0.235151] devtmpfs: initialized [ 0.236214] x86/mm: Memory block size: 128MB [ 0.238359] gcov: version magic: 0x41383552 [ 0.241431] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.244082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.247459] pinctrl core: initialized pinctrl subsystem [ 0.249164] [ 0.249654] ************************************************************* [ 0.251011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253010] ** ** [ 0.256011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.258011] ** ** [ 0.260011] ** This means that this kernel is built to expose internal ** [ 0.262009] ** IOMMU data structures, which may compromise security on ** [ 0.264010] ** your system. ** [ 0.266008] ** ** [ 0.268008] ** If you see this message and you are not debugging the ** [ 0.270008] ** kernel, report this immediately to your vendor! ** [ 0.272009] ** ** [ 0.275008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277008] ************************************************************* [ 0.279046] NET: Registered protocol family 16 [ 0.281513] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.283048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.286046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.288416] cpuidle: using governor menu [ 0.290556] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.293482] PCI: Using configuration type 1 for base access [ 0.294101] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.303122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.305054] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.308172] cryptd: max_cpu_qlen set to 1000 [ 0.311436] ACPI: Added _OSI(Module Device) [ 0.312007] ACPI: Added _OSI(Processor Device) [ 0.312809] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.314010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.316761] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.323424] ACPI: Interpreter enabled [ 0.324070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.326019] ACPI: Using IOAPIC for interrupt routing [ 0.327143] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.329399] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.338159] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.340038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.343018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.345074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.349126] acpiphp: Slot [2] registered [ 0.350113] acpiphp: Slot [3] registered [ 0.351010] acpiphp: Slot [4] registered [ 0.351814] acpiphp: Slot [5] registered [ 0.352063] acpiphp: Slot [6] registered [ 0.353069] acpiphp: Slot [7] registered [ 0.353986] acpiphp: Slot [8] registered [ 0.355099] acpiphp: Slot [9] registered [ 0.355944] acpiphp: Slot [10] registered [ 0.357043] acpiphp: Slot [11] registered [ 0.358071] acpiphp: Slot [12] registered [ 0.359057] acpiphp: Slot [13] registered [ 0.360058] acpiphp: Slot [14] registered [ 0.361082] acpiphp: Slot [15] registered [ 0.362056] acpiphp: Slot [16] registered [ 0.363054] acpiphp: Slot [17] registered [ 0.364055] acpiphp: Slot [18] registered [ 0.365058] acpiphp: Slot [19] registered [ 0.366062] acpiphp: Slot [20] registered [ 0.368114] acpiphp: Slot [21] registered [ 0.369067] acpiphp: Slot [22] registered [ 0.370061] acpiphp: Slot [23] registered [ 0.372066] acpiphp: Slot [24] registered [ 0.373060] acpiphp: Slot [25] registered [ 0.374064] acpiphp: Slot [26] registered [ 0.375065] acpiphp: Slot [27] registered [ 0.377093] acpiphp: Slot [28] registered [ 0.378066] acpiphp: Slot [29] registered [ 0.380066] acpiphp: Slot [30] registered [ 0.381076] acpiphp: Slot [31] registered [ 0.382054] PCI host bridge to bus 0000:00 [ 0.384015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.386011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.387009] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.390016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.392010] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.394014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.396195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.399580] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.402474] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.407308] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.409393] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.412014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.413013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.415013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.417549] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.419739] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.422033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.424652] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.428015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.435963] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.441027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.446138] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.454097] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.464036] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.478014] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.487526] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.493015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.501017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.520029] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.531434] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.534370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.537590] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.540330] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.542196] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.547100] iommu: Default domain type: Passthrough [ 0.549500] SCSI subsystem initialized [ 0.550159] ACPI: bus type USB registered [ 0.552099] usbcore: registered new interface driver usbfs [ 0.553070] usbcore: registered new interface driver hub [ 0.555100] usbcore: registered new device driver usb [ 0.557222] pps_core: LinuxPPS API ver. 1 registered [ 0.559011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.561071] PTP clock support registered [ 0.562129] EDAC MC: Ver: 3.0.0 [ 0.563362] PCI: Using ACPI for IRQ routing [ 0.565701] NetLabel: Initializing [ 0.566008] NetLabel: domain hash size = 128 [ 0.567009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.569295] NetLabel: unlabeled traffic allowed by default [ 0.572235] vgaarb: loaded [ 0.573237] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.575011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.579674] clocksource: Switched to clocksource kvm-clock [ 0.683603] VFS: Disk quotas dquot_6.6.0 [ 0.685268] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.688282] *** VALIDATE ramfs *** [ 0.689786] *** VALIDATE hugetlbfs *** [ 0.691588] pnp: PnP ACPI init [ 0.694428] pnp: PnP ACPI: found 6 devices [ 0.714187] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.717150] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.719071] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.720778] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.722974] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.725153] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.727484] NET: Registered protocol family 2 [ 0.729821] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.734185] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.737366] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.742096] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.745096] TCP: Hash tables configured (established 65536 bind 65536) [ 0.747567] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.750136] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.752632] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.754966] NET: Registered protocol family 1 [ 0.757137] RPC: Registered named UNIX socket transport module. [ 0.759011] RPC: Registered udp transport module. [ 0.760416] RPC: Registered tcp transport module. [ 0.761722] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.763531] NET: Registered protocol family 44 [ 0.764781] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.766494] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.768148] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.770118] PCI: CLS 0 bytes, default 64 [ 0.772224] Unpacking initramfs... [ 2.311159] debug: unmapping init [mem 0xffff8dc57cc64000-0xffff8dc57ffcffff] [ 2.315086] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.317066] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.320145] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.860052] Initialise system trusted keyrings [ 2.861675] Key type blacklist registered [ 2.865824] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.874919] zbud: loaded [ 2.878095] *** VALIDATE nfs *** [ 2.879283] *** VALIDATE nfs4 *** [ 2.880846] pstore: using deflate compression [ 2.885804] Platform Keyring initialized [ 3.156847] NET: Registered protocol family 38 [ 3.158547] Key type asymmetric registered [ 3.165197] Asymmetric key parser 'x509' registered [ 3.168659] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.173039] io scheduler mq-deadline registered [ 3.174540] io scheduler kyber registered [ 3.176045] io scheduler bfq registered [ 3.179242] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.187987] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.190709] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.194343] ACPI: Power Button [PWRF] [ 3.295904] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.495493] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.700272] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.762013] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.837219] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.864740] Non-volatile memory driver v1.3 [ 3.867078] Linux agpgart interface v0.103 [ 4.013326] virtio_blk virtio1: [vda] 132968 512-byte logical blocks (68.1 MB/64.9 MiB) [ 4.024575] vda: detected capacity change from 0 to 68079616 [ 4.083188] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.085165] vdb: detected capacity change from 0 to 1073741824 [ 4.101097] libphy: Fixed MDIO Bus: probed [ 4.139320] usbcore: registered new interface driver usbserial_generic [ 4.143628] usbserial: USB Serial support registered for generic [ 4.151565] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.164844] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.166731] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.171385] mousedev: PS/2 mouse device common for all mice [ 4.184056] rtc_cmos 00:05: RTC can wake from S4 [ 4.196672] rtc_cmos 00:05: registered as rtc0 [ 4.197649] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.200478] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.203267] intel_pstate: CPU model not supported [ 4.230793] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.234633] hid: raw HID events driver (C) Jiri Kosina [ 4.239082] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.247955] usbcore: registered new interface driver usbhid [ 4.249928] usbhid: USB HID core driver [ 4.251370] drop_monitor: Initializing network drop monitor service [ 4.256822] Initializing XFRM netlink socket [ 4.258721] NET: Registered protocol family 10 [ 4.263774] Segment Routing with IPv6 [ 4.265423] NET: Registered protocol family 17 [ 4.268541] mpls_gso: MPLS GSO support [ 4.274943] RAS: Correctable Errors collector initialized. [ 4.277092] AVX version of gcm_enc/dec engaged. [ 4.278688] AES CTR mode by8 optimization enabled [ 4.592255] sched_clock: Marking stable (4592233421, 0)->(5293409168, -701175747) [ 4.605575] registered taskstats version 1 [ 4.613265] Loading compiled-in X.509 certificates [ 4.618662] zswap: loaded using pool lzo/zbud [ 4.689451] Key type big_key registered [ 4.717544] Key type encrypted registered [ 4.719519] ima: No TPM chip found, activating TPM-bypass! [ 4.725507] ima: Allocated hash algorithm: sha1 [ 4.729149] ima: No architecture policies found [ 4.732939] evm: Initialising EVM extended attributes: [ 4.736469] evm: security.selinux [ 4.738331] evm: security.ima [ 4.739520] evm: security.capability [ 4.741274] evm: HMAC attrs: 0x1 [ 4.749988] rtc_cmos 00:05: setting system clock to 2025-08-13 14:37:10 UTC (1755095830) [ 4.765645] debug: unmapping init [mem 0xffffffffae803000-0xffffffffae9fffff] [ 4.770287] debug: unmapping init [mem 0xffffffffad582000-0xffffffffad858fff] [ 4.785423] Write protecting the kernel read-only data: 28672k [ 4.790676] debug: unmapping init [mem 0xffffffffabc03000-0xffffffffabdfffff] [ 4.793960] debug: unmapping init [mem 0xffffffffac514000-0xffffffffac5fffff] [ 4.862300] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.875730] systemd[1]: Detected virtualization kvm. [ 4.881820] systemd[1]: Detected architecture x86-64. [ 4.886339] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.930368] systemd[1]: No hostname configured. [ 4.936325] systemd[1]: Set hostname to . [ 4.938246] random: systemd: uninitialized urandom read (16 bytes read) [ 4.944416] systemd[1]: Initializing machine ID from random generator. [ 5.326392] random: systemd: uninitialized urandom read (16 bytes read) [ 5.330604] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.343647] random: systemd: uninitialized urandom read (16 bytes read) [ 5.348875] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.356928] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.923492] device-mapper: uevent: version 1.0.3 [ 6.925714] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.903049] virtio_net virtio0 ens2: renamed from eth0 [ 9.632811] scsi host0: ata_piix [ 9.649918] scsi host1: ata_piix [ 9.651295] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 9.653710] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.374182] random: crng init done [ 15.375442] random: 7 urandom warning(s) missed due to ratelimiting [ 18.724360] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.926501] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.470906] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.219339] SELinux: Disabled at runtime. [ 23.303915] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.319783] systemd[1]: Detected virtualization kvm. [ 23.322808] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.676240] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.688391] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.706996] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.710747] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.717447] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.753240] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.776165] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 25.261372] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.203145] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.386646] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.457603] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.057194] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.384653] EDAC sbridge: Ver: 1.1.2 [ 31.832658] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.753838] NFS: Registering the id_resolver key type [ 32.756355] Key type id_resolver registered [ 32.758187] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg207-client login: [ 55.963028] hrtimer: interrupt took 8020725 ns [ 79.955397] libcfs: loading out-of-tree module taints kernel. [ 79.991450] Key type ._llcrypt registered [ 80.000319] Key type .llcrypt registered [ 80.386289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 80.394604] alg: No test for adler32 (adler32-zlib) [ 81.486188] Lustre: Lustre: Build Version: 2.16.56_9_g5b8a063 [ 81.916112] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 81.918660] LNet: Accept secure, port 988 [ 83.559287] Key type lgssc registered [ 84.149535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.909378] Lustre: Mounted lustre-client [ 140.393379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 154.677566] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing check_logdir /tmp/testlogs/ [ 156.380284] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing yml_node [ 158.732156] Lustre: DEBUG MARKER: Client: 2.16.56.9 [ 159.865938] Lustre: DEBUG MARKER: MDS: 2.16.56.9 [ 161.084507] Lustre: DEBUG MARKER: OSS: 2.16.56.9 [ 161.975440] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 13 10:39:46 EDT 2025 [ 163.808305] Lustre: lustre-OST0000-osc-ffff8dc5d1187800: disconnect after 24s idle [ 167.717885] LustreError: 7932:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8dc5d1187800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 167.722656] LustreError: 7932:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 167.734085] Lustre: Unmounted lustre-client [ 194.285219] Key type lgssc unregistered [ 194.444598] LNet: 8380:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 195.493760] LNet: Removed LNI 192.168.202.7@tcp [ 195.824043] Key type .llcrypt unregistered [ 195.825622] Key type ._llcrypt unregistered [ 207.391034] Key type ._llcrypt registered [ 207.393027] Key type .llcrypt registered [ 207.497137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 207.504645] alg: No test for adler32 (adler32-zlib) [ 208.327818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 208.362284] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 208.364904] LNet: Accept secure, port 988 [ 208.374925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 210.089807] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 210.155550] LNet: 9247:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 210.165612] LNet: Removed LNI 192.168.202.7@tcp [ 210.466182] Key type .llcrypt unregistered [ 210.467827] Key type ._llcrypt unregistered [ 212.890118] Key type ._llcrypt registered [ 212.892365] Key type .llcrypt registered [ 213.473201] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:40:38 (1755096038) [ 213.616180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 213.623050] alg: No test for adler32 (adler32-zlib) [ 214.461754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.498971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.510284] LNet: 9655:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 214.528105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.556879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 214.568467] LNet: 9662:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 217.602456] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:40:42 (1755096042) [ 217.641376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.656623] LNet: 10231:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 217.668074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.715324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 220.618339] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:40:45 (1755096045) [ 220.654318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.667511] LNet: 10817:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 220.686247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.730117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 223.513262] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:40:48 (1755096048) [ 223.544619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.561316] LNet: 11400:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 223.572519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.613074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 226.393794] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:40:51 (1755096051) [ 226.466029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.485231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.524096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 226.711221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 229.520833] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:40:54 (1755096054) [ 229.575903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.588179] LNet: 12576:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 229.591529] LNet: 12576:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 229.604790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.639668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 229.829498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 230.030438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 233.135830] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:40:58 (1755096058) [ 233.196214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.227425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.263565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 236.178666] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:41:01 (1755096061) [ 236.243459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.267473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.303565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 236.319188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 236.519082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 236.540561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 236.763439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 236.781558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 236.989979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 237.009714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 237.205717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 237.230661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 237.445813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 237.472485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 237.664581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 237.686164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 237.914692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 237.932393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 238.153950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 238.177872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 238.399862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 238.416640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 238.619849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 238.637649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 238.845581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 238.869543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 239.069447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 239.087532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 242.293080] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:41:07 (1755096067) [ 242.350758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 242.362134] LNet: 14437:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 242.365368] LNet: 14437:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 242.379934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.421115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 242.439529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 245.799993] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:41:10 (1755096070) [ 245.862207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 245.887479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 245.965170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 245.988662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 249.241855] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:41:14 (1755096074) [ 249.288691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.317880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.357600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 249.376907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 252.700201] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:41:17 (1755096077) [ 252.810828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.835876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.873872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 252.891722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 255.859709] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:41:20 (1755096080) [ 255.910625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.939415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.972677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 255.993670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 259.025581] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:41:24 (1755096084) [ 259.059935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 259.071786] LNet: 17361:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.075030] LNet: 17361:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 259.087175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 259.133273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 259.156376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 262.031045] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:41:27 (1755096087) [ 262.068828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.086767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.127097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 262.309503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 265.031580] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:41:30 (1755096090) [ 265.085446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.116547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.222985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 265.329890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 268.347516] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:41:33 (1755096093) [ 268.418623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.442627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.501643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.599539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 271.837952] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:41:36 (1755096096) [ 271.949746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.971693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.172351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 272.285213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 275.316769] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:41:40 (1755096100) [ 275.380783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.418131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.467591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 275.497729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 278.517387] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:41:43 (1755096103) [ 278.627445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 278.651208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 278.690909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 278.722643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 281.752624] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:41:46 (1755096106) [ 281.809384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 281.828521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 281.880204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 284.800863] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:41:49 (1755096109) [ 284.851850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 284.871492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.906950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 287.722646] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:41:52 (1755096112) [ 287.797059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.820821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.868979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 288.068812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 291.132559] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:41:56 (1755096116) [ 291.246618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 291.259384] LNet: 23387:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 291.262957] LNet: 23387:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 291.278883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.316737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 291.341761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 294.375818] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:41:59 (1755096119) [ 294.438603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.467579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.508940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 294.526976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 297.282168] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:42:02 (1755096122) [ 297.342282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.363828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.409787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 297.436599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 300.314253] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:42:05 (1755096125) [ 300.378868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.400350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.453860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 300.475208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 303.270496] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:42:08 (1755096128) [ 303.311352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.333236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.366656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 303.398270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 303.408948] LNetError: 25742:0:(peer.c:2185:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 303.426040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 303.441935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 305.891838] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:42:11 (1755096131) [ 305.938163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.959557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.002332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.iOjslXe2c8 [ 306.038755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.iOjslXe2c8 [ 306.065743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.iOjslXe2c8 [ 308.561852] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:42:13 (1755096133) [ 308.628612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.655886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.681654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 308.698952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 308.720643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 311.338559] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:42:16 (1755096136) [ 311.403051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.423692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.454849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 311.472539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 311.490686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 311.508174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 311.524223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 311.539959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 311.552850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 311.568309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 311.585218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 311.603759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 311.618891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 311.633943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 311.650980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 311.668652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 311.685949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 311.704284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 311.727556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 311.749928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 311.769363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 311.781686] LNetError: 27541:0:(router.c:632:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 314.345581] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:42:19 (1755096139) [ 314.408577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.428851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.462601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 317.542821] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:42:22 (1755096142) [ 318.409178] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg207-server [ 319.206581] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:42:24 (1755096144) [ 319.297574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.318538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.354344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 319.368104] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 319.370534] LNet: Accept secure, port 988 [ 320.112350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[8-12]@tcp [ 320.125442] LNet: 29252:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 320.148857] LNetError: 29045:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.202.10@tcp is being used as a gateway but routing feature is not turned on [ 320.158055] LNetError: 29045:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.10@tcp has gone from up to down [ 322.888205] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:42:28 (1755096148) [ 322.913533] LNetError: 29835:0:(router.c:334:lnet_is_route_alive()) peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 323.231359] LNetError: 29049:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.12@tcp has gone from up to down [ 323.244931] LNetError: 29049:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 323.700829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg207-server [ 324.339722] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:42:29 (1755096149) [ 324.375058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.414620] LNet: Removed LNI 192.168.202.7@tcp [ 325.431800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.460440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 325.479990] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 325.483173] LNet: Accept secure, port 988 [ 325.758994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[8-13/2]@tcp [ 325.771576] LNet: 30301:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 325.777295] LNet: 30301:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 325.789167] LNetError: 30190:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.202.8@tcp is being used as a gateway but routing feature is not turned on [ 325.791869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[8-13/2]@tcp [ 325.796040] LNetError: 30190:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 325.810878] LNetError: 30190:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.8@tcp has gone from up to down [ 325.816120] LNetError: 30190:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 328.784628] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:42:33 (1755096153) [ 328.864318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.896648] LNet: Removed LNI 192.168.202.7@tcp [ 329.909424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.950450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 329.981376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.002496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.039528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.073735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.096431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.129378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 330.170934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.190827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.226438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.270292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.296637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.324653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 330.360319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.378209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.414167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.457949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.480918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.513185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 330.550893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.577755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.620554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.661542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.683263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.709414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 330.746332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.766513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.796874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.833662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.852611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.879205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 330.909262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.930283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.972686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 333.755511] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:42:38 (1755096158) [ 334.572183] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg207-server [ 335.312794] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:42:40 (1755096160) [ 336.115728] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg207-server [ 336.809529] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:42:42 (1755096162) [ 337.152709] Key type .llcrypt unregistered [ 337.154180] Key type ._llcrypt unregistered [ 339.462924] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 339.523968] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 339.527745] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 339.531780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 339.662629] Key type ._llcrypt registered [ 339.664293] Key type .llcrypt registered [ 339.763172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 339.770745] alg: No test for adler32 (adler32-zlib) [ 340.599079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.632543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 340.647785] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 340.650543] LNet: Accept secure, port 988 [ 340.970344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 340.982517] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 341.936793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 341.947779] LNet: Removed LNI 10.1.2.3@tcp [ 342.000356] LNet: 32981:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.012840] LNet: Removed LNI 192.168.202.7@tcp [ 342.309118] Key type .llcrypt unregistered [ 342.310772] Key type ._llcrypt unregistered [ 347.017909] Key type ._llcrypt registered [ 347.019396] Key type .llcrypt registered [ 347.688577] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:42:52 (1755096172) [ 347.911506] Key type .llcrypt unregistered [ 347.913162] Key type ._llcrypt unregistered [ 350.534401] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 350.565938] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 350.568487] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 350.571237] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 350.767190] Key type ._llcrypt registered [ 350.771259] Key type .llcrypt registered [ 350.889224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 350.900391] alg: No test for adler32 (adler32-zlib) [ 351.719947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.753666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 351.770523] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 351.774317] LNet: Accept secure, port 988 [ 352.152743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 352.165865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 353.398855] LNet: 34436:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.404496] LNet: Removed LNI 192.168.202.7@tcp [ 353.416052] LNet: Removed LNI 10.1.2.3@tcp [ 353.735613] Key type .llcrypt unregistered [ 353.737447] Key type ._llcrypt unregistered [ 358.349215] Key type ._llcrypt registered [ 358.350997] Key type .llcrypt registered [ 358.984396] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:43:04 (1755096184) [ 359.250325] Key type .llcrypt unregistered [ 359.252132] Key type ._llcrypt unregistered [ 361.611289] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 361.663818] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 361.666239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 361.669429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 361.809831] Key type ._llcrypt registered [ 361.811319] Key type .llcrypt registered [ 361.892348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 361.900783] alg: No test for adler32 (adler32-zlib) [ 362.724485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.755734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 362.784957] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 362.788701] LNet: Accept secure, port 988 [ 363.750959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 363.762195] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 364.917374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 364.928046] LNet: Removed LNI 10.1.2.3@tcp [ 364.936191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 364.956176] LNet: Removed LNI 10.1.2.31@tcp [ 364.965881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 364.980720] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 365.310375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 365.320187] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 365.711698] LNet: 36312:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.715311] LNet: Removed LNI 10.1.2.3@tcp [ 365.723498] LNet: Removed LNI 10.1.2.31@tcp [ 366.027590] Key type .llcrypt unregistered [ 366.029124] Key type ._llcrypt unregistered [ 370.442910] Key type ._llcrypt registered [ 370.444487] Key type .llcrypt registered [ 371.003859] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:43:16 (1755096196) [ 371.194418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 371.201340] alg: No test for adler32 (adler32-zlib) [ 372.023949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.054692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 372.068658] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 372.072197] LNet: Accept secure, port 988 [ 372.407931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.419837] LNet: 37256:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.431348] LNet: Removed LNI 192.168.202.7@tcp [ 372.441511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.470532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.7@tcp [ 372.484164] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 372.486352] LNet: Accept secure, port 988 [ 372.836706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.7@tcp [ 372.859059] LNet: Removed LNI 192.168.202.7@tcp [ 373.153462] Key type .llcrypt unregistered [ 373.155217] Key type ._llcrypt unregistered [ 378.025325] Key type ._llcrypt registered [ 378.026428] Key type .llcrypt registered [ 378.725586] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:43:23 (1755096203) [ 379.597600] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg207-server [ 380.463533] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:43:25 (1755096205) [ 380.708478] Key type .llcrypt unregistered [ 380.710208] Key type ._llcrypt unregistered [ 383.195759] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 383.297604] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 383.301399] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 383.304098] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 383.404397] Key type ._llcrypt registered [ 383.406257] Key type .llcrypt registered [ 383.505349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 383.515896] alg: No test for adler32 (adler32-zlib) [ 384.350367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.376465] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 384.378819] LNet: Accept secure, port 988 [ 384.705970] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 385.762224] LNet: 39185:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.765947] LNet: Removed LNI 192.168.202.7@tcp1 [ 385.774351] LNet: Removed LNI 10.1.2.3@tcp2 [ 386.068428] Key type .llcrypt unregistered [ 386.070208] Key type ._llcrypt unregistered [ 391.091498] Key type ._llcrypt registered [ 391.092826] Key type .llcrypt registered [ 391.711039] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:43:36 (1755096216) [ 391.858480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 391.865148] alg: No test for adler32 (adler32-zlib) [ 392.691982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.728116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 392.740543] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 392.743309] LNet: Accept secure, port 988 [ 393.088628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.7 [ 393.102953] LNet: Removed LNI 192.168.202.7@tcp [ 395.856180] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:43:40 (1755096220) [ 395.919544] LNet: 40707:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.938243] LNet: Removed LNI 192.168.202.7@tcp [ 396.284779] Key type .llcrypt unregistered [ 396.287881] Key type ._llcrypt unregistered [ 398.825725] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 398.944501] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 398.946871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 398.949636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 399.140609] Key type ._llcrypt registered [ 399.142258] Key type .llcrypt registered [ 399.221779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 399.235075] alg: No test for adler32 (adler32-zlib) [ 400.107427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 400.110370] LNet: Accept secure, port 988 [ 403.229644] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:43:48 (1755096228) [ 403.280897] LNet: 41570:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.293484] LNet: Removed LNI 10.1.2.3@tcp [ 403.618260] Key type .llcrypt unregistered [ 403.620100] Key type ._llcrypt unregistered [ 406.438881] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 406.567298] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 406.569482] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 406.571661] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 406.740569] Key type ._llcrypt registered [ 406.742046] Key type .llcrypt registered [ 406.820985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 406.828768] alg: No test for adler32 (adler32-zlib) [ 407.699685] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 407.703225] LNet: Accept secure, port 988 [ 411.283978] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:43:56 (1755096236) [ 411.331787] LNet: 42435:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.346119] LNet: Removed LNI 10.1.2.3@tcp [ 411.650825] Key type .llcrypt unregistered [ 411.652234] Key type ._llcrypt unregistered [ 414.130913] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 414.235893] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 414.238258] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 414.242584] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 414.404367] Key type ._llcrypt registered [ 414.406260] Key type .llcrypt registered [ 414.495217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 414.506364] alg: No test for adler32 (adler32-zlib) [ 415.337162] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 415.338755] LNet: Accept secure, port 988 [ 418.157492] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:44:03 (1755096243) [ 418.199616] LNet: 43298:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.207981] LNet: Removed LNI 10.1.2.3@tcp [ 418.574055] Key type .llcrypt unregistered [ 418.575679] Key type ._llcrypt unregistered [ 421.145279] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 421.283628] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 421.285722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 421.288467] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 421.445912] Key type ._llcrypt registered [ 421.447490] Key type .llcrypt registered [ 421.549589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 421.557922] alg: No test for adler32 (adler32-zlib) [ 422.396868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.435111] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 422.438736] LNet: Accept secure, port 988 [ 426.133160] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:44:11 (1755096251) [ 426.360445] LNet: 44506:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.369984] LNet: Removed LNI 10.1.2.3@tcp [ 426.688940] Key type .llcrypt unregistered [ 426.690513] Key type ._llcrypt unregistered [ 429.153160] Key type ._llcrypt registered [ 429.155659] Key type .llcrypt registered [ 429.241593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 429.249103] alg: No test for adler32 (adler32-zlib) [ 430.077139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 430.093132] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 430.096102] LNet: Accept secure, port 988 [ 433.336641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 433.601893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_interrupt [ 433.623299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_interrupt [ 433.644897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 433.655788] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 433.787894] LNet: 44918:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 433.831422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 433.850206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 433.871945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 434.115297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_dropped [ 434.135643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_dropped [ 434.156756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 434.174991] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 434.299384] LNet: 45003:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 434.335109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 434.355026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 434.373165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 434.580619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_aborted [ 434.602299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_aborted [ 434.623896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 434.795773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 434.817117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 434.837762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 435.042426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_no_route [ 435.060912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_no_route [ 435.079980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 435.225564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 435.242806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 435.258862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 435.466255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_timeout [ 435.488079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_timeout [ 435.507222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 435.516550] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 435.519886] LNet: Skipped 2 previous similar messages [ 435.627938] LNet: 45258:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 435.630548] LNet: 45258:0:(api-ni.c:337:recovery_interval_set()) Skipped 2 previous similar messages [ 435.664325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 435.681947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 435.699251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 435.924917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_error [ 435.943965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_error [ 435.963518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 436.133610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 436.148994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 436.165250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 436.286513] LNet: 45379:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.356514] LNet: Removed LNI 192.168.202.7@tcp [ 437.579574] Key type .llcrypt unregistered [ 437.580991] Key type ._llcrypt unregistered [ 443.869572] Key type ._llcrypt registered [ 443.871295] Key type .llcrypt registered [ 444.444212] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:44:29 (1755096269) [ 444.794437] Key type .llcrypt unregistered [ 444.796060] Key type ._llcrypt unregistered [ 447.016148] Key type ._llcrypt registered [ 447.017773] Key type .llcrypt registered [ 447.099871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 447.107523] alg: No test for adler32 (adler32-zlib) [ 447.926228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 447.939461] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 447.941557] LNet: Accept secure, port 988 [ 451.119830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 454.797757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.813215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 454.823426] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 455.392878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_interrupt [ 455.410655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp1 -m GET -r 1 -e local_interrupt [ 455.429668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_interrupt [ 455.450917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -m GET -r 1 -e local_interrupt [ 455.469820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp -m GET -r 1 -e local_interrupt [ 455.486810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp1 -m GET -r 1 -e local_interrupt [ 455.505866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -m GET -r 1 -e local_interrupt [ 455.524321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -m GET -r 1 -e local_interrupt [ 455.542030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 455.551951] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 455.555628] LNet: 1 local NIs in recovery (showing 1): 192.168.202.7@tcp [ 455.685473] LNet: 46812:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 455.720229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 455.739577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 455.756577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 455.960054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_dropped [ 455.983092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp1 -m GET -r 1 -e local_dropped [ 456.002366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_dropped [ 456.023712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -m GET -r 1 -e local_dropped [ 456.042559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp -m GET -r 1 -e local_dropped [ 456.063873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp1 -m GET -r 1 -e local_dropped [ 456.081622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -m GET -r 1 -e local_dropped [ 456.101162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -m GET -r 1 -e local_dropped [ 456.118458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 456.133536] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 456.139442] LNet: Skipped 2 previous similar messages [ 456.247586] LNet: 46910:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 456.278565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 456.294532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 456.310277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 456.516745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_aborted [ 456.534217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp1 -m GET -r 1 -e local_aborted [ 456.553933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_aborted [ 456.573994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -m GET -r 1 -e local_aborted [ 456.594377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp -m GET -r 1 -e local_aborted [ 456.614620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp1 -m GET -r 1 -e local_aborted [ 456.633327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -m GET -r 1 -e local_aborted [ 456.652216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -m GET -r 1 -e local_aborted [ 456.670857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 456.838470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 456.856682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 456.873983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 457.068741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_no_route [ 457.086642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp1 -m GET -r 1 -e local_no_route [ 457.106107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_no_route [ 457.125097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -m GET -r 1 -e local_no_route [ 457.143555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp -m GET -r 1 -e local_no_route [ 457.162488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp1 -m GET -r 1 -e local_no_route [ 457.182133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -m GET -r 1 -e local_no_route [ 457.200561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -m GET -r 1 -e local_no_route [ 457.216483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 457.228137] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 457.233332] LNet: Skipped 6 previous similar messages [ 457.336748] LNet: 47104:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 457.339353] LNet: 47104:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 457.372809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 457.390194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 457.407393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 457.639400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_timeout [ 457.657875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp1 -m GET -r 1 -e local_timeout [ 457.675275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_timeout [ 457.692793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -m GET -r 1 -e local_timeout [ 457.712576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp -m GET -r 1 -e local_timeout [ 457.730469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp1 -m GET -r 1 -e local_timeout [ 457.749087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -m GET -r 1 -e local_timeout [ 457.768741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -m GET -r 1 -e local_timeout [ 457.786914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 457.949892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 457.967690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 457.983842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 458.193689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e local_error [ 458.212517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp1 -m GET -r 1 -e local_error [ 458.232045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e local_error [ 458.250538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -m GET -r 1 -e local_error [ 458.269120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp -m GET -r 1 -e local_error [ 458.287094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp1 -m GET -r 1 -e local_error [ 458.307904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -m GET -r 1 -e local_error [ 458.328768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -m GET -r 1 -e local_error [ 458.348905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 458.511983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 458.533343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 458.549608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 460.840923] LNet: 47368:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.855270] LNet: Removed LNI 192.168.202.7@tcp [ 462.885539] LNet: Removed LNI 192.168.202.7@tcp1 [ 463.066811] Key type .llcrypt unregistered [ 463.068351] Key type ._llcrypt unregistered [ 469.192169] Key type ._llcrypt registered [ 469.193551] Key type .llcrypt registered [ 469.737111] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:44:54 (1755096294) [ 470.060474] Key type .llcrypt unregistered [ 470.061801] Key type ._llcrypt unregistered [ 472.237703] Key type ._llcrypt registered [ 472.239585] Key type .llcrypt registered [ 472.312519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 472.320343] alg: No test for adler32 (adler32-zlib) [ 473.137265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 473.151335] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 473.153387] LNet: Accept secure, port 988 [ 476.302800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 476.568218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e remote_dropped [ 476.590157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e remote_dropped [ 476.608848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 476.620260] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 476.742561] LNet: 48589:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 476.777285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 476.792581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 476.808142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 477.021704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e remote_error [ 477.041289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e remote_error [ 477.060273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 477.236721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 477.253778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 477.270963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 477.472658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e remote_timeout [ 477.491110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e remote_timeout [ 477.509441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 477.519579] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 477.522575] LNet: Skipped 1 previous similar message [ 477.639325] LNet: 48759:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 477.642401] LNet: 48759:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 477.678088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 477.696673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 477.713692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 477.833041] LNet: 48795:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 477.841701] LNet: Removed LNI 192.168.202.7@tcp [ 478.044835] Key type .llcrypt unregistered [ 478.046496] Key type ._llcrypt unregistered [ 484.266941] Key type ._llcrypt registered [ 484.268202] Key type .llcrypt registered [ 484.795724] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:45:09 (1755096309) [ 485.148553] Key type .llcrypt unregistered [ 485.149940] Key type ._llcrypt unregistered [ 487.209496] Key type ._llcrypt registered [ 487.211226] Key type .llcrypt registered [ 487.286187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 487.293363] alg: No test for adler32 (adler32-zlib) [ 488.111846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 488.124947] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 488.127437] LNet: Accept secure, port 988 [ 491.231950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 494.373631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.387926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 494.397543] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 494.907642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e remote_dropped [ 494.925372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp1 -m GET -r 1 -e remote_dropped [ 494.942425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e remote_dropped [ 494.959367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -m GET -r 1 -e remote_dropped [ 494.977592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp -m GET -r 1 -e remote_dropped [ 494.994267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp1 -m GET -r 1 -e remote_dropped [ 495.012970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -m GET -r 1 -e remote_dropped [ 495.031870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -m GET -r 1 -e remote_dropped [ 495.051188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 495.062073] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 495.065709] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.107@tcp [ 495.170970] LNet: 50228:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 495.204029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 495.220210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 495.236796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 495.253263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 495.444513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e remote_error [ 495.462286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp1 -m GET -r 1 -e remote_error [ 495.481042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e remote_error [ 495.500796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -m GET -r 1 -e remote_error [ 495.519797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp -m GET -r 1 -e remote_error [ 495.538930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp1 -m GET -r 1 -e remote_error [ 495.558811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -m GET -r 1 -e remote_error [ 495.577301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -m GET -r 1 -e remote_error [ 495.595454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 495.608867] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 495.614026] LNet: Skipped 2 previous similar messages [ 495.720966] LNet: 50327:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 495.752709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 495.768276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 495.783206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 495.798509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 495.979585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e remote_timeout [ 495.998199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp1 -m GET -r 1 -e remote_timeout [ 496.017473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e remote_timeout [ 496.038212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -m GET -r 1 -e remote_timeout [ 496.059537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp -m GET -r 1 -e remote_timeout [ 496.080409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp1 -m GET -r 1 -e remote_timeout [ 496.099527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -m GET -r 1 -e remote_timeout [ 496.117251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -m GET -r 1 -e remote_timeout [ 496.135743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.107@tcp [ 496.277966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 496.294837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 496.313528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 496.331174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 497.570506] LNet: 50499:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 498.591407] LNet: Removed LNI 192.168.202.7@tcp [ 498.599157] LNet: Removed LNI 192.168.202.7@tcp1 [ 498.775505] Key type .llcrypt unregistered [ 498.777253] Key type ._llcrypt unregistered [ 504.463020] Key type ._llcrypt registered [ 504.464631] Key type .llcrypt registered [ 504.971338] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:45:30 (1755096330) [ 505.158303] Key type .llcrypt unregistered [ 505.159893] Key type ._llcrypt unregistered [ 507.322178] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 507.369525] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 507.372298] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 507.374936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 507.517901] Key type ._llcrypt registered [ 507.519754] Key type .llcrypt registered [ 507.597671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 507.607262] alg: No test for adler32 (adler32-zlib) [ 508.423887] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 508.426343] LNet: Accept secure, port 988 [ 508.443991] LNet: 51560:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 508.459978] LNet: Removed LNI 192.168.202.7@tcp [ 508.740395] Key type .llcrypt unregistered [ 508.742181] Key type ._llcrypt unregistered [ 510.878356] Key type ._llcrypt registered [ 510.880039] Key type .llcrypt registered [ 510.966541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 510.974308] alg: No test for adler32 (adler32-zlib) [ 511.791449] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 511.793790] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 511.796059] LNet: Accept secure, port 988 [ 511.810919] LNet: 51828:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 511.814690] LNet: Removed LNI 192.168.202.7@tcp [ 511.821858] LNet: Removed LNI 10.1.2.3@tcp1 [ 512.096390] Key type .llcrypt unregistered [ 512.097521] Key type ._llcrypt unregistered [ 514.208259] Key type ._llcrypt registered [ 514.209854] Key type .llcrypt registered [ 514.280951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 514.288589] alg: No test for adler32 (adler32-zlib) [ 515.109933] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 515.112666] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 515.116239] LNet: Accept secure, port 988 [ 515.137876] LNet: 52097:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 515.140793] LNet: Removed LNI 192.168.202.7@tcp [ 515.149763] LNet: Removed LNI 10.1.2.3@tcp [ 515.464871] Key type .llcrypt unregistered [ 515.466283] Key type ._llcrypt unregistered [ 517.727975] Key type ._llcrypt registered [ 517.729724] Key type .llcrypt registered [ 517.801567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 517.810607] alg: No test for adler32 (adler32-zlib) [ 518.632937] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 518.634662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 518.636795] LNet: Accept secure, port 988 [ 518.653651] LNet: 52367:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.656750] LNet: Removed LNI 192.168.202.7@tcp [ 518.663557] LNet: Removed LNI 10.1.2.3@tcp [ 518.954372] Key type .llcrypt unregistered [ 518.956096] Key type ._llcrypt unregistered [ 521.275790] Key type ._llcrypt registered [ 521.277461] Key type .llcrypt registered [ 521.349331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 521.356663] alg: No test for adler32 (adler32-zlib) [ 522.176030] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 522.177609] LNetError: 52626:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 522.180623] LNetError: Error -100 starting up LNI tcp [ 522.189421] LNet: Removed LNI 192.168.202.7@tcp [ 522.476325] Key type .llcrypt unregistered [ 522.477662] Key type ._llcrypt unregistered [ 526.771805] Key type ._llcrypt registered [ 526.773155] Key type .llcrypt registered [ 527.357257] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:45:52 (1755096352) [ 527.720335] Key type .llcrypt unregistered [ 527.721854] Key type ._llcrypt unregistered [ 529.936484] Key type ._llcrypt registered [ 529.938097] Key type .llcrypt registered [ 530.019351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 530.026451] alg: No test for adler32 (adler32-zlib) [ 530.839537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 530.854944] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 530.857653] LNet: Accept secure, port 988 [ 533.943039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 534.178574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e network_timeout [ 534.198140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e network_timeout [ 534.216534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 534.226490] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 534.329938] LNet: 53853:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 534.370868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 534.388850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 534.499810] LNet: 53887:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 534.508541] LNet: Removed LNI 192.168.202.7@tcp [ 534.673260] Key type .llcrypt unregistered [ 534.675040] Key type ._llcrypt unregistered [ 540.822557] Key type ._llcrypt registered [ 540.824130] Key type .llcrypt registered [ 540.903689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 540.911209] alg: No test for adler32 (adler32-zlib) [ 541.723799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 541.737240] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 541.739672] LNet: Accept secure, port 988 [ 544.969133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 548.240357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.260486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 548.270358] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 548.800301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp -m GET -r 1 -e network_timeout [ 548.816886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.107@tcp1 -m GET -r 1 -e network_timeout [ 548.836385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp -m GET -r 1 -e network_timeout [ 548.854525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -m GET -r 1 -e network_timeout [ 548.871704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp -m GET -r 1 -e network_timeout [ 548.891076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.107@tcp1 -m GET -r 1 -e network_timeout [ 548.912348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -m GET -r 1 -e network_timeout [ 548.932789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -m GET -r 1 -e network_timeout [ 548.953140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 548.964213] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 549.066274] LNet: 54734:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 549.095261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 549.115726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 551.292123] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 551.402565] LNet: 54803:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.415164] LNet: Removed LNI 192.168.202.7@tcp [ 552.422416] LNet: Removed LNI 192.168.202.7@tcp1 [ 552.593515] Key type .llcrypt unregistered [ 552.595203] Key type ._llcrypt unregistered [ 558.277394] Key type ._llcrypt registered [ 558.278933] Key type .llcrypt registered [ 558.802339] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:46:23 (1755096383) [ 558.969539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 558.977294] alg: No test for adler32 (adler32-zlib) [ 559.790926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.816813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 559.828884] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 559.832339] LNet: Accept secure, port 988 [ 560.111478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 560.120917] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 560.445106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.7@tcp [ 560.476090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 560.499272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 560.516791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 560.537800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 560.555585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 560.573135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.7@tcp [ 560.584026] LNetError: 55892:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.202.7@tcp to recovery [ 560.863480] LNet: 1 local NIs in recovery (showing 1): 192.168.202.7@tcp [ 560.863544] LNet: There was an unexpected network error while writing to 192.168.202.7: rc = -22 [ 562.911847] LNet: There was an unexpected network error while writing to 192.168.202.7: rc = -22 [ 563.757138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 567.007921] LNet: There was an unexpected network error while writing to 192.168.202.7: rc = -22 [ 567.975088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 567.993904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 568.020649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.031348] LNet: 56009:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.035643] LNet: Removed LNI 192.168.202.7@tcp [ 568.053084] LNet: Removed LNI 192.168.202.7@tcp1 [ 568.062657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.086315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 568.097830] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 568.100276] LNet: Accept secure, port 988 [ 568.396686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 568.408462] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 568.776705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.7@tcp [ 568.795127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 568.825458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 568.848672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 568.865347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 568.882625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 568.899620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 568.916488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.7@tcp [ 568.929454] LNetError: 56275:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.202.7@tcp to recovery [ 569.119511] LNet: There was an unexpected network error while writing to 192.168.202.7: rc = -22 [ 572.119525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 575.264384] LNet: There was an unexpected network error while writing to 192.168.202.7: rc = -22 [ 575.267989] LNet: Skipped 1 previous similar message [ 579.452670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 579.468516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 579.485615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 582.226813] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:46:47 (1755096407) [ 582.268320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.278986] LNet: 56987:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.327191] LNet: Removed LNI 192.168.202.7@tcp [ 583.335601] LNet: Removed LNI 192.168.202.7@tcp1 [ 583.345461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.371612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 583.384887] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 583.387510] LNet: Accept secure, port 988 [ 583.704911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 583.715360] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 584.076730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.7@tcp [ 584.108171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 584.124311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 584.142039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 584.162045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 584.181387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 584.198782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.7@tcp --health 0 [ 584.241488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 584.415195] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.7@tcp [ 585.439289] LNet: There was an unexpected network error while writing to 192.168.202.7: rc = -22 [ 585.442645] LNet: Skipped 1 previous similar message [ 600.772295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 600.836646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 600.852643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 600.869632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.7@tcp --health 500 [ 600.916370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 607.175831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 607.203105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.213454] LNet: 57493:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.224090] LNet: Removed LNI 192.168.202.7@tcp [ 608.235132] LNet: Removed LNI 192.168.202.7@tcp1 [ 608.248324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.277816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 608.293679] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 608.296655] LNet: Accept secure, port 988 [ 608.616833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 608.629144] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 609.022745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.7@tcp [ 609.042908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 609.071765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 609.088325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 609.111456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 609.131196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 609.148893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 609.164680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.7@tcp --health 0 [ 610.335398] LNet: There was an unexpected network error while writing to 192.168.202.7: rc = -22 [ 610.339653] LNet: Skipped 3 previous similar messages [ 611.345969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 620.876708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 620.896585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 620.916080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 624.149093] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:47:29 (1755096449) [ 624.356084] LNet: 58543:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.375152] LNet: Removed LNI 192.168.202.7@tcp [ 625.384852] LNet: Removed LNI 192.168.202.7@tcp1 [ 625.711885] Key type .llcrypt unregistered [ 625.713521] Key type ._llcrypt unregistered [ 628.460259] Key type ._llcrypt registered [ 628.461670] Key type .llcrypt registered [ 628.559101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 628.570614] alg: No test for adler32 (adler32-zlib) [ 629.401232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 629.416065] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 629.418614] LNet: Accept secure, port 988 [ 633.941880] Lustre: DEBUG MARKER: Initial discovery [ 633.963556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.107@tcp [ 635.029285] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 635.055521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 635.075710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.107@tcp [ 635.088698] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 635.100857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 635.119582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.107@tcp -r 1 -m GET -l 3 [ 635.138422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.107@tcp -r 1 -m GET -e local_error [ 635.157218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.107@tcp -r 1 -m PUT -l 6 [ 635.793627] Lustre: DEBUG MARKER: Force oleg207-client.virtnet to discover 192.168.202.107@tcp (in background) [ 635.851833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.107@tcp [ 636.496823] Lustre: DEBUG MARKER: Force oleg207-server to discover 192.168.202.7@tcp [ 637.489517] Lustre: DEBUG MARKER: Wait for 59295 [ 638.879890] LNet: There was an unexpected network error while writing to 192.168.202.107: rc = -22 [ 639.557460] Lustre: DEBUG MARKER: Finished wait on 59295 [ 639.599790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 639.624888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 639.755585] LNet: 59746:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.805441] LNet: Removed LNI 192.168.202.7@tcp [ 640.997862] Key type .llcrypt unregistered [ 641.000940] Key type ._llcrypt unregistered [ 647.210621] Key type ._llcrypt registered [ 647.211691] Key type .llcrypt registered [ 647.762674] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:47:52 (1755096472) [ 647.981284] Key type .llcrypt unregistered [ 647.982813] Key type ._llcrypt unregistered [ 650.139126] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 650.191779] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 650.194268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 650.196237] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 650.348540] Key type ._llcrypt registered [ 650.353872] Key type .llcrypt registered [ 650.435627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 650.443492] alg: No test for adler32 (adler32-zlib) [ 651.255224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.284664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.297514] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 651.299984] LNet: Accept secure, port 988 [ 651.624479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 651.635460] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 654.868260] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:48:00 (1755096480) [ 654.906805] LNet: 61779:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.910884] LNet: Removed LNI 192.168.202.7@tcp [ 654.917795] LNet: Removed LNI 10.1.2.3@tcp [ 655.179599] Key type .llcrypt unregistered [ 655.181062] Key type ._llcrypt unregistered [ 657.458166] Key type ._llcrypt registered [ 657.459796] Key type .llcrypt registered [ 657.539174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 657.547532] alg: No test for adler32 (adler32-zlib) [ 658.359220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.385261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 658.398256] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 658.401121] LNet: Accept secure, port 988 [ 658.686965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 658.696145] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 659.079952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.7@tcp [ 660.127199] LNetError: Unexpected error -22 connecting to 192.168.202.7@tcp at host 192.168.202.7:988 [ 661.683704] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:48:06 (1755096486) [ 661.740869] LNet: 62897:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.751155] LNet: Removed LNI 192.168.202.7@tcp [ 662.759171] LNet: Removed LNI 10.1.2.3@tcp [ 663.034455] Key type .llcrypt unregistered [ 663.036219] Key type ._llcrypt unregistered [ 665.350230] Key type ._llcrypt registered [ 665.351597] Key type .llcrypt registered [ 665.433162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 665.440664] alg: No test for adler32 (adler32-zlib) [ 666.269039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.298659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 666.313752] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 666.316453] LNet: Accept secure, port 988 [ 666.639757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 666.652110] LNet: Added LNI 192.168.202.7@tcp2 [8/256/0/180] [ 667.045560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.7@tcp1 --nid 192.168.202.7@tcp2 [ 669.155120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.7@tcp1 192.168.202.7@tcp2 [ 669.167485] LNetError: 63725:0:(lib-move.c:2225:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.7@tcp1 to 192.168.202.7@tcp2 - Route table may be misconfigured [ 669.180921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.7@tcp2 192.168.202.7@tcp1 [ 672.085347] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:48:17 (1755096497) [ 672.134543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.149737] LNet: 64294:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.183217] LNet: Removed LNI 192.168.202.7@tcp1 [ 674.213687] LNet: Removed LNI 192.168.202.7@tcp2 [ 674.223828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.251329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.264408] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 674.267130] LNet: Accept secure, port 988 [ 674.555463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 674.566764] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 674.920514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.7@tcp [ 674.940604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.7@tcp 192.168.202.7@tcp [ 674.957786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.7@tcp1 192.168.202.7@tcp1 [ 674.976685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.7@tcp -d 192.168.202.7@tcp -e network_timeout [ 674.994276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.7@tcp -d 192.168.202.7@tcp1 -e network_timeout [ 675.013508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp -e network_timeout [ 675.032561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.7@tcp1 -d 192.168.202.7@tcp1 -e network_timeout [ 675.051226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 675.063329] LNet: There was an unexpected network error while writing to 192.168.202.7: rc = -22 [ 675.073145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 675.120510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 675.161153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 675.295121] LNet: 1 local NIs in recovery (showing 1): 192.168.202.7@tcp [ 677.693591] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:48:22 (1755096502) [ 677.725658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.737486] LNet: 65149:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.751240] LNet: Removed LNI 192.168.202.7@tcp [ 679.781819] LNet: Removed LNI 192.168.202.7@tcp1 [ 679.792885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.832352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 679.945630] LNet: 65191:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.124597] Key type .llcrypt unregistered [ 680.126337] Key type ._llcrypt unregistered [ 684.514695] Key type ._llcrypt registered [ 684.516381] Key type .llcrypt registered [ 685.084616] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:48:30 (1755096510) [ 685.223104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.230964] alg: No test for adler32 (adler32-zlib) [ 686.056844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.696432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 687.415681] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:48:32 (1755096512) [ 687.463225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.474264] LNet: 66281:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.486193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.516944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.530848] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 687.533252] LNet: Accept secure, port 988 [ 687.826939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 687.838353] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 688.202460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 688.220604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp1 [ 688.241519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.7@tcp1 [ 690.865294] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:48:36 (1755096516) [ 691.678412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg207-server [ 692.466234] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:48:37 (1755096517) [ 693.241990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg207-server [ 693.957227] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:48:39 (1755096519) [ 694.696865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg207-server [ 695.378105] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:48:40 (1755096520) [ 696.102581] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg207-server [ 696.759195] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:48:41 (1755096521) [ 697.446642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg207-server [ 698.080431] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:48:43 (1755096523) [ 698.765334] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg207-server [ 699.463731] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:48:44 (1755096524) [ 700.248924] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg207-server [ 700.968264] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:48:46 (1755096526) [ 701.731992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg207-server [ 702.448752] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:48:47 (1755096527) [ 702.513124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.524624] LNet: 69812:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.583217] LNet: Removed LNI 192.168.202.7@tcp [ 704.614965] LNet: Removed LNI 192.168.202.7@tcp1 [ 704.625226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.654662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.669955] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 704.676195] LNet: Accept secure, port 988 [ 705.027396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 705.064651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 705.144136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.162547] LNet: 69958:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.213392] LNet: Removed LNI 192.168.202.7@tcp [ 706.224731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.259018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 706.280832] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 706.284671] LNet: Accept secure, port 988 [ 706.631613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 706.705196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.749650] LNet: Removed LNI 192.168.202.7@tcp [ 707.762221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.790824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 707.806301] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 707.809183] LNet: Accept secure, port 988 [ 708.119275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 708.152767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 708.237434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.247971] LNet: 70244:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.251448] LNet: 70244:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 709.285783] LNet: Removed LNI 192.168.202.7@tcp [ 709.297266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.326688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 709.339489] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 709.341972] LNet: Accept secure, port 988 [ 709.673630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 709.770599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.821385] LNet: Removed LNI 192.168.202.7@tcp [ 710.830504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.860426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 710.873643] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 710.876611] LNet: Accept secure, port 988 [ 711.189444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 711.228126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 711.322577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.358071] LNet: Removed LNI 192.168.202.7@tcp [ 712.369671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.396122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 712.410468] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 712.412864] LNet: Accept secure, port 988 [ 712.756256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 712.870691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.881729] LNet: 70673:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.884585] LNet: 70673:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 713.957404] LNet: Removed LNI 192.168.202.7@tcp [ 713.966806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.993374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 714.006453] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 714.009080] LNet: Accept secure, port 988 [ 714.329823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 714.369061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 714.457671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.494044] LNet: Removed LNI 192.168.202.7@tcp [ 715.508980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.536221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 715.555383] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 715.558126] LNet: Accept secure, port 988 [ 715.901572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 716.049610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.093554] LNet: Removed LNI 192.168.202.7@tcp [ 717.103312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.132168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.144907] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 717.147673] LNet: Accept secure, port 988 [ 717.491914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 717.534687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 717.663626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.695309] LNet: Removed LNI 192.168.202.7@tcp [ 718.706074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.733323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 718.749452] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 718.752703] LNet: Accept secure, port 988 [ 719.127625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 719.275066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.294818] LNet: Removed LNI 192.168.202.7@tcp [ 720.306345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.342188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 720.355622] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 720.359118] LNet: Accept secure, port 988 [ 720.668212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 720.708349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 720.822932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.893880] LNet: Removed LNI 192.168.202.7@tcp [ 721.912832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.941749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 721.956589] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 721.961427] LNet: Accept secure, port 988 [ 722.307659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 722.461518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.473104] LNet: 71533:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.476137] LNet: 71533:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 723.494518] LNet: Removed LNI 192.168.202.7@tcp [ 723.508210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.554827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 723.571900] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 723.574970] LNet: Accept secure, port 988 [ 723.970624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 723.998377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.024808] LNet: Removed LNI 192.168.202.7@tcp [ 724.034289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.064305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 724.079561] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 724.082423] LNet: Accept secure, port 988 [ 724.441284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 727.483586] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:49:12 (1755096552) [ 727.515833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.535095] LNet: Removed LNI 192.168.202.7@tcp [ 727.547380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.576855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 727.589989] LNet: Added LNI 192.168.202.7@tcp231 [8/256/0/180] [ 727.593735] LNet: Accept secure, port 988 [ 728.109902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.127461] LNet: Removed LNI 192.168.202.7@tcp231 [ 728.136863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.169724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 728.181781] LNet: Added LNI 192.168.202.7@tcp231 [8/256/0/180] [ 728.184391] LNet: Accept secure, port 988 [ 728.186644] LNet: 72506:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 728.371391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 728.392408] LNet: Removed LNI 192.168.202.7@tcp231 [ 728.399956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 728.411368] LNet: Added LNI 192.168.202.7@tcp231 [8/256/0/180] [ 728.883078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.900654] LNet: Removed LNI 192.168.202.7@tcp231 [ 728.910410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.947862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 728.962163] LNet: Added LNI 192.168.202.7@tcp231 [8/256/0/0] [ 728.964848] LNet: Accept secure, port 988 [ 728.967230] LNet: 72655:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 729.141792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 729.163369] LNet: Removed LNI 192.168.202.7@tcp231 [ 729.173263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 729.187909] LNet: Added LNI 192.168.202.7@tcp231 [8/256/0/0] [ 729.653736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.671635] LNet: Removed LNI 192.168.202.7@tcp231 [ 729.681063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.712787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 729.727548] LNet: Added LNI 192.168.202.7@tcp231 [8/256/0/60] [ 729.729664] LNet: Accept secure, port 988 [ 729.896301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 729.916343] LNet: Removed LNI 192.168.202.7@tcp231 [ 729.924832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 729.937487] LNet: Added LNI 192.168.202.7@tcp231 [8/256/0/60] [ 730.408277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.425368] LNet: Removed LNI 192.168.202.7@tcp231 [ 730.436406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.469186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 730.482521] LNet: Added LNI 192.168.202.7@tcp231 [8/256/0/180] [ 730.485290] LNet: Accept secure, port 988 [ 730.487295] LNet: 72951:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 730.489981] LNet: 72951:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 730.669452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 730.683963] LNet: Removed LNI 192.168.202.7@tcp231 [ 730.692348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 730.705828] LNet: Added LNI 192.168.202.7@tcp231 [8/256/0/180] [ 733.597993] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:49:18 (1755096558) [ 733.634450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 733.651709] LNet: Removed LNI 192.168.202.7@tcp231 [ 733.662516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.698969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 733.711851] LNet: Added LNI 192.168.202.7@tcp232 [8/256/0/180] [ 733.714423] LNet: Accept secure, port 988 [ 733.987781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 734.004419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 734.020608] LNet: Removed LNI 192.168.202.7@tcp232 [ 734.028822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 734.039073] LNet: Added LNI 192.168.202.7@tcp232 [8/256/0/180] [ 734.322706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 734.342845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 734.359747] LNet: Removed LNI 192.168.202.7@tcp232 [ 734.368241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 734.381214] LNet: Added LNI 192.168.202.7@tcp232 [8/256/0/180] [ 734.679750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 734.697189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 734.716633] LNet: Removed LNI 192.168.202.7@tcp232 [ 734.724834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 734.734857] LNet: Added LNI 192.168.202.7@tcp232 [8/256/0/180] [ 735.034497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 735.052143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 735.067340] LNet: Removed LNI 192.168.202.7@tcp232 [ 735.075892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 737.455875] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:49:22 (1755096562) [ 737.498739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.518874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.548305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.561862] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 737.564764] LNet: Accept secure, port 988 [ 737.857382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 737.886232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 737.907128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 737.940380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 737.962113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 737.982290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 740.447351] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:49:25 (1755096565) [ 740.490678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.501643] LNet: 75449:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.505196] LNet: 75449:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 740.517090] LNet: Removed LNI 192.168.202.7@tcp [ 740.528663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.554770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.568146] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 740.570577] LNet: Accept secure, port 988 [ 743.243556] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:49:28 (1755096568) [ 743.810244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 744.465685] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:49:29 (1755096569) [ 744.629706] LNet: Removed LNI 192.168.202.7@tcp [ 744.861080] Key type .llcrypt unregistered [ 744.862144] Key type ._llcrypt unregistered [ 747.033956] Key type ._llcrypt registered [ 747.035910] Key type .llcrypt registered [ 747.119149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 747.126465] alg: No test for adler32 (adler32-zlib) [ 747.948235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 747.965768] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 747.968893] LNet: Accept secure, port 988 [ 751.317452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 754.125210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.107@tcp [ 754.266325] LNet: 76893:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.277604] LNet: Removed LNI 192.168.202.7@tcp [ 754.464451] Key type .llcrypt unregistered [ 754.465965] Key type ._llcrypt unregistered [ 758.704775] Key type ._llcrypt registered [ 758.706169] Key type .llcrypt registered [ 759.256162] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 759.895149] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 760.603293] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:49:45 (1755096585) [ 760.736880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 760.744326] alg: No test for adler32 (adler32-zlib) [ 761.584022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.630196] LNet: 77975:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.922766] Key type .llcrypt unregistered [ 761.924476] Key type ._llcrypt unregistered [ 764.390867] Key type ._llcrypt registered [ 764.392877] Key type .llcrypt registered [ 764.494232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.502389] alg: No test for adler32 (adler32-zlib) [ 765.331687] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 765.334121] LNet: 78227:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 765.339388] LNet: Accept secure, port 988 [ 765.356364] LNetError: 78239:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 765.364075] LNetError: 78239:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.202.11@tcp has gone from up to down [ 765.446884] LNetError: 78258:0:(router.c:334:lnet_is_route_alive()) peer 192.168.202.10@tcp is being used as a gateway but routing feature is not turned on [ 765.576779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.671181] LNetError: 78239:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.202.9@tcp has gone from up to down [ 768.676588] LNetError: 78239:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 768.680792] LNet: 78283:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.702116] LNet: Removed LNI 192.168.202.7@tcp [ 769.716146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 769.725582] LNet: 78285:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 772.453572] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:49:57 (1755096597) [ 773.237534] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg207-server [ 773.963605] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:49:59 (1755096599) [ 774.626689] Lustre: Lustre: Build Version: 2.16.56_9_g5b8a063 [ 774.733387] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 774.735418] LNet: 79294:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 774.739415] LNet: 79294:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 774.742386] LNet: Accept secure, port 988 [ 774.754162] LNetError: 79309:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 774.763303] LNetError: 79309:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 774.768995] LNetError: 79309:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.202.11@tcp has gone from up to down [ 774.773500] LNetError: 79309:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 776.351910] Key type lgssc registered [ 776.859624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 777.824702] LNetError: 79299:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.202.9@tcp has gone from up to down [ 777.831128] LNetError: 79299:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 785.910785] Key type lgssc unregistered [ 786.049492] LNet: 80330:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.109528] LNet: Removed LNI 192.168.202.7@tcp [ 787.393746] Key type .llcrypt unregistered [ 787.395126] Key type ._llcrypt unregistered [ 792.964266] Key type ._llcrypt registered [ 792.965867] Key type .llcrypt registered [ 793.531850] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:50:18 (1755096618) [ 793.657908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 793.664523] alg: No test for adler32 (adler32-zlib) [ 794.555537] LNetError: 81169:0:(api-ni.c:2619:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 794.711595] Key type .llcrypt unregistered [ 794.713179] Key type ._llcrypt unregistered [ 796.749921] Key type ._llcrypt registered [ 796.751540] Key type .llcrypt registered [ 797.298856] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:50:22 (1755096622) [ 797.547420] Key type .llcrypt unregistered [ 797.549083] Key type ._llcrypt unregistered [ 799.967282] Key type ._llcrypt registered [ 799.968910] Key type .llcrypt registered [ 800.094853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.104735] alg: No test for adler32 (adler32-zlib) [ 803.695034] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:50:28 (1755096628) [ 803.733040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.751249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.783339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.798142] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 803.801037] LNet: Accept secure, port 988 [ 804.140285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.157590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 804.172636] LNet: Removed LNI 192.168.202.7@tcp [ 804.276352] LNet: 82745:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.446709] Key type .llcrypt unregistered [ 804.448471] Key type ._llcrypt unregistered [ 808.642935] Key type ._llcrypt registered [ 808.644015] Key type .llcrypt registered [ 809.161449] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:50:34 (1755096634) [ 809.289311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 809.295206] alg: No test for adler32 (adler32-zlib) [ 810.107916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.139134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.152691] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 810.155813] LNet: Accept secure, port 988 [ 810.453362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 810.470994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.7@tcp [ 810.481105] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 810.483153] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 810.485154] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 810, 0, 0 [ 810.487547] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 810.489022] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 810.491194] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 810.493188] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 810, 0, 0 [ 810.495690] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 810.497141] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 810.498982] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 810.500890] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 810, 0, 860 [ 810.503354] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 810.505147] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 810.507482] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 810.509422] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 810, 0, 860 [ 810.512187] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 813.018508] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:50:38 (1755096638) [ 813.189885] LNet: 84284:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.246492] LNet: Removed LNI 192.168.202.7@tcp [ 814.516667] Key type .llcrypt unregistered [ 814.519133] Key type ._llcrypt unregistered [ 816.642264] Key type ._llcrypt registered [ 816.643940] Key type .llcrypt registered [ 816.717989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 816.726047] alg: No test for adler32 (adler32-zlib) [ 817.537755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 817.551820] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 817.554323] LNet: Accept secure, port 988 [ 820.801368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 823.901945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.917124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 823.927086] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 824.339375] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 824.341855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 824.376038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 824.387980] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 828.847483] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:50:54 (1755096654) [ 828.884916] LNet: 85737:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.919369] LNet: Removed LNI 192.168.202.7@tcp [ 830.943140] LNet: Removed LNI 192.168.202.7@tcp1 [ 830.953690] LNet: Removed LNI 10.1.2.3@tcp99 [ 831.232451] Key type .llcrypt unregistered [ 831.233946] Key type ._llcrypt unregistered [ 833.658395] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 833.661212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 833.766215] Key type ._llcrypt registered [ 833.767769] Key type .llcrypt registered [ 833.866119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.875187] alg: No test for adler32 (adler32-zlib) [ 834.699087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.727675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.739301] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 834.741442] LNet: Accept secure, port 988 [ 835.065955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 835.077977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 836.030415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 836.051576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 836.072552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 836.083305] LNetError: 86456:0:(peer.c:2185:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 836.096704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 836.126687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.7@tcp [ 836.331830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 836.366446] LNet: 86476:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.407140] LNet: Removed LNI 192.168.202.7@tcp [ 837.414416] LNet: Removed LNI 10.1.2.3@tcp [ 837.687110] Key type .llcrypt unregistered [ 837.688695] Key type ._llcrypt unregistered [ 841.980406] Key type ._llcrypt registered [ 841.981647] Key type .llcrypt registered [ 842.499988] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:51:07 (1755096667) [ 842.628353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.635169] alg: No test for adler32 (adler32-zlib) [ 843.448732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.475573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.487799] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 843.490111] LNet: Accept secure, port 988 [ 843.780323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.7@tcp [ 843.816936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg207-client@tcp [ 846.262512] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:51:11 (1755096671) [ 846.318501] LNet: 87994:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.333897] LNet: Removed LNI 192.168.202.7@tcp [ 847.620778] Key type .llcrypt unregistered [ 847.622345] Key type ._llcrypt unregistered [ 849.860058] Key type ._llcrypt registered [ 849.861587] Key type .llcrypt registered [ 849.929194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 849.935814] alg: No test for adler32 (adler32-zlib) [ 850.744856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 850.756992] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 850.759125] LNet: Accept secure, port 988 [ 850.784503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 853.345451] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:51:18 (1755096678) [ 853.931873] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 854.672873] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:51:19 (1755096679) [ 855.276920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 855.956866] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:51:21 (1755096681) [ 856.028592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.040794] LNet: 89413:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.049772] LNet: Removed LNI 192.168.202.7@tcp [ 856.059277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.086530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 856.102442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 856.374279] Key type .llcrypt unregistered [ 856.375940] Key type ._llcrypt unregistered [ 860.308325] Key type ._llcrypt registered [ 860.309560] Key type .llcrypt registered [ 860.876913] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:51:26 (1755096686) [ 861.022197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 861.031185] alg: No test for adler32 (adler32-zlib) [ 861.842291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.869221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.881397] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 861.883406] LNet: Accept secure, port 988 [ 862.168648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 862.193581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.7@tcp [ 864.689307] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:51:29 (1755096689) [ 864.745391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.757696] LNet: 90957:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.830560] LNet: Removed LNI 192.168.202.7@tcp [ 865.839278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.863913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 865.877735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 868.297869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 868.934718] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:51:34 (1755096694) [ 868.970735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.981550] LNet: 91674:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.994213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.080278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 869.092348] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 869.095196] LNet: Accept secure, port 988 [ 869.353195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 869.362372] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 869.671654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.7@tcp [ 869.688345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 869.713710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.7@tcp -m PUT -e local_timeout -r 1 [ 869.738361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 869.761587] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 869.764267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 869.846195] LNetError: 91702:0:(lib-move.c:4909:lnet_parse()) 192.168.202.7@tcp, src 192.168.202.7@tcp: Dropping PUT (error -108 looking up sender) [ 870.879404] LNet: 91978:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.903189] LNet: Removed LNI 192.168.202.7@tcp [ 871.915488] LNet: Removed LNI 10.1.2.3@tcp [ 872.090525] Key type .llcrypt unregistered [ 872.092113] Key type ._llcrypt unregistered [ 876.257501] Key type ._llcrypt registered [ 876.259164] Key type .llcrypt registered [ 876.839850] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:51:41 (1755096701) [ 876.998860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.005904] alg: No test for adler32 (adler32-zlib) [ 877.839294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.005068] LNet: 92827:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.263425] Key type .llcrypt unregistered [ 878.265071] Key type ._llcrypt unregistered [ 880.592562] Key type ._llcrypt registered [ 880.593964] Key type .llcrypt registered [ 880.683740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 880.693488] alg: No test for adler32 (adler32-zlib) [ 881.510388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 881.526568] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 881.529529] LNet: Accept secure, port 988 [ 884.811519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 884.876846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 885.074757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.107@tcp [ 885.193277] LNet: 93251:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 885.226909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 885.244683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 885.480217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 885.617980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 885.632805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 885.860702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.107@tcp [ 885.974382] LNet: 93437:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 885.977561] LNet: 93437:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 886.012556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 886.027849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 886.256293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 886.405927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 886.423385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 886.553690] LNet: 93571:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.563372] LNet: Removed LNI 192.168.202.7@tcp [ 886.730598] Key type .llcrypt unregistered [ 886.732159] Key type ._llcrypt unregistered [ 892.789968] Key type ._llcrypt registered [ 892.791362] Key type .llcrypt registered [ 893.334879] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:51:58 (1755096718) [ 893.481450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 893.487715] alg: No test for adler32 (adler32-zlib) [ 894.304213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.460747] LNet: 94454:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.720494] Key type .llcrypt unregistered [ 894.721669] Key type ._llcrypt unregistered [ 897.103470] Key type ._llcrypt registered [ 897.105167] Key type .llcrypt registered [ 897.197312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.205886] alg: No test for adler32 (adler32-zlib) [ 898.044240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 898.058123] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 898.062079] LNet: Accept secure, port 988 [ 901.468023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.107@tcp [ 904.649552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.666881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 904.677673] LNet: Added LNI 192.168.202.7@tcp1 [8/256/0/180] [ 905.082120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 905.309878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.107@tcp [ 905.431561] LNet: 95077:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 905.465851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 905.482798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 905.746583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 905.899853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 905.915110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 906.205074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.107@tcp [ 906.271360] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.107@tcp [ 906.334544] LNet: 95265:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 906.337720] LNet: 95265:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 906.371429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 906.388683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 906.663605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 906.819736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 906.836987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 909.176466] LNet: 95433:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.239184] LNet: Removed LNI 192.168.202.7@tcp [ 910.247435] LNet: Removed LNI 192.168.202.7@tcp1 [ 910.433079] Key type .llcrypt unregistered [ 910.434308] Key type ._llcrypt unregistered [ 916.569530] Key type ._llcrypt registered [ 916.571055] Key type .llcrypt registered [ 917.139492] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 755 sec =========== 10:52:22 (1755096742) [ 917.408341] Key type .llcrypt unregistered [ 917.410070] Key type ._llcrypt unregistered [ 921.949404] Key type ._llcrypt registered [ 921.950913] Key type .llcrypt registered [ 922.132677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.141231] alg: No test for adler32 (adler32-zlib) [ 923.029952] Lustre: Lustre: Build Version: 2.16.56_9_g5b8a063 [ 923.139073] LNet: Added LNI 192.168.202.7@tcp [8/256/0/180] [ 923.141481] LNet: Accept secure, port 988 [ 924.743256] Key type lgssc registered [ 925.298763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 951.521180] Lustre: Mounted lustre-client [ 954.292542] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 962.068917] LustreError: 98623:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8dc5d0481000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 962.078532] LustreError: 98623:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 962.094169] Lustre: Unmounted lustre-client [ 975.090241] Key type lgssc unregistered [ 975.230608] LNet: 99100:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.294355] LNet: Removed LNI 192.168.202.7@tcp [ 976.595939] Key type .llcrypt unregistered [ 976.597538] Key type ._llcrypt unregistered