[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 376203226 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002248] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006013] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007005] pid_max: default: 32768 minimum: 301 [ 0.008100] LSM: Security Framework initializing [ 0.009028] Yama: becoming mindful. [ 0.010021] SELinux: Initializing. [ 0.011022] *** VALIDATE selinux *** [ 0.017592] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021217] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022081] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023059] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024077] *** VALIDATE tmpfs *** [ 0.025357] *** VALIDATE proc *** [ 0.026172] *** VALIDATE cgroup *** [ 0.027006] *** VALIDATE cgroup2 *** [ 0.028215] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029107] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031019] Spectre V2 : User space: Vulnerable [ 0.032003] Speculative Store Bypass: Vulnerable [ 0.035308] debug: unmapping init [mem 0xffffffffb2a59000-0xffffffffb2a60fff] [ 0.037872] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038427] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039010] ... version: 2 [ 0.039835] ... bit width: 48 [ 0.040006] ... generic registers: 4 [ 0.040833] ... value mask: 0000ffffffffffff [ 0.041005] ... max period: 00007fffffffffff [ 0.042005] ... fixed-purpose events: 3 [ 0.042810] ... event mask: 000000070000000f [ 0.043260] rcu: Hierarchical SRCU implementation. [ 0.045249] smp: Bringing up secondary CPUs ... [ 0.046393] x86: Booting SMP configuration: [ 0.047016] .... node #0, CPUs: #1 #2 #3 [ 0.050432] smp: Brought up 1 node, 4 CPUs [ 0.051816] smpboot: Max logical packages: 1 [ 0.052008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.279395] node 0 deferred pages initialised in 226ms [ 0.281569] devtmpfs: initialized [ 0.282230] x86/mm: Memory block size: 128MB [ 0.284270] gcov: version magic: 0x41383552 [ 0.287106] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.289079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.290240] pinctrl core: initialized pinctrl subsystem [ 0.292090] [ 0.292627] ************************************************************* [ 0.294007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.295005] ** ** [ 0.297006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.298005] ** ** [ 0.300006] ** This means that this kernel is built to expose internal ** [ 0.301006] ** IOMMU data structures, which may compromise security on ** [ 0.302006] ** your system. ** [ 0.304006] ** ** [ 0.305004] ** If you see this message and you are not debugging the ** [ 0.306006] ** kernel, report this immediately to your vendor! ** [ 0.308006] ** ** [ 0.309005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.310006] ************************************************************* [ 0.313791] NET: Registered protocol family 16 [ 0.315499] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.318040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.320042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.324064] cpuidle: using governor menu [ 0.325576] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.328137] PCI: Using configuration type 1 for base access [ 0.329131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.337133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.339028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.342097] cryptd: max_cpu_qlen set to 1000 [ 0.344241] ACPI: Added _OSI(Module Device) [ 0.345008] ACPI: Added _OSI(Processor Device) [ 0.346011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.347015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.349782] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.353416] ACPI: Interpreter enabled [ 0.354825] ACPI: PM: (supports S0 S3 S4 S5) [ 0.355008] ACPI: Using IOAPIC for interrupt routing [ 0.355949] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.358246] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.365169] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.366019] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.368009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.370056] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.372931] acpiphp: Slot [2] registered [ 0.373133] acpiphp: Slot [3] registered [ 0.373934] acpiphp: Slot [4] registered [ 0.375045] acpiphp: Slot [5] registered [ 0.375920] acpiphp: Slot [6] registered [ 0.377073] acpiphp: Slot [7] registered [ 0.377920] acpiphp: Slot [8] registered [ 0.379078] acpiphp: Slot [9] registered [ 0.379862] acpiphp: Slot [10] registered [ 0.380045] acpiphp: Slot [11] registered [ 0.380889] acpiphp: Slot [12] registered [ 0.382050] acpiphp: Slot [13] registered [ 0.382930] acpiphp: Slot [14] registered [ 0.384058] acpiphp: Slot [15] registered [ 0.384881] acpiphp: Slot [16] registered [ 0.386059] acpiphp: Slot [17] registered [ 0.386963] acpiphp: Slot [18] registered [ 0.387048] acpiphp: Slot [19] registered [ 0.387884] acpiphp: Slot [20] registered [ 0.389047] acpiphp: Slot [21] registered [ 0.389896] acpiphp: Slot [22] registered [ 0.391054] acpiphp: Slot [23] registered [ 0.391852] acpiphp: Slot [24] registered [ 0.392044] acpiphp: Slot [25] registered [ 0.392870] acpiphp: Slot [26] registered [ 0.394046] acpiphp: Slot [27] registered [ 0.394881] acpiphp: Slot [28] registered [ 0.396046] acpiphp: Slot [29] registered [ 0.396879] acpiphp: Slot [30] registered [ 0.397047] acpiphp: Slot [31] registered [ 0.397881] PCI host bridge to bus 0000:00 [ 0.399011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.400011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.401008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.403009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.405013] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.407015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.408174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.412470] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.417399] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.424010] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.428049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.430011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.433011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.435009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.437495] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.441071] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.444037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.447740] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.452009] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.463018] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.468015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.473122] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.478019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.481013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.492016] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.504587] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.512012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.519013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.531019] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.537507] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.539210] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.541217] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.543195] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.544109] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.548131] iommu: Default domain type: Passthrough [ 0.549292] SCSI subsystem initialized [ 0.550072] ACPI: bus type USB registered [ 0.550958] usbcore: registered new interface driver usbfs [ 0.552034] usbcore: registered new interface driver hub [ 0.554034] usbcore: registered new device driver usb [ 0.555090] pps_core: LinuxPPS API ver. 1 registered [ 0.556005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.558027] PTP clock support registered [ 0.560029] EDAC MC: Ver: 3.0.0 [ 0.561097] PCI: Using ACPI for IRQ routing [ 0.562518] NetLabel: Initializing [ 0.563006] NetLabel: domain hash size = 128 [ 0.563926] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.564073] NetLabel: unlabeled traffic allowed by default [ 0.566098] vgaarb: loaded [ 0.567239] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.568008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.575324] clocksource: Switched to clocksource kvm-clock [ 0.660101] VFS: Disk quotas dquot_6.6.0 [ 0.661567] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.664024] *** VALIDATE ramfs *** [ 0.665267] *** VALIDATE hugetlbfs *** [ 0.666759] pnp: PnP ACPI init [ 0.669199] pnp: PnP ACPI: found 6 devices [ 0.683864] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.686294] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.687561] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.688788] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.690164] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.691448] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.693024] NET: Registered protocol family 2 [ 0.694581] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.697838] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.699922] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.703685] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.705735] TCP: Hash tables configured (established 65536 bind 65536) [ 0.707458] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.709249] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.710953] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.712755] NET: Registered protocol family 1 [ 0.714310] RPC: Registered named UNIX socket transport module. [ 0.715483] RPC: Registered udp transport module. [ 0.716427] RPC: Registered tcp transport module. [ 0.717386] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.718638] NET: Registered protocol family 44 [ 0.719545] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.720818] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.722043] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.723334] PCI: CLS 0 bytes, default 64 [ 0.724368] Unpacking initramfs... [ 1.903567] debug: unmapping init [mem 0xffff8a53fcc64000-0xffff8a53fffcffff] [ 1.907305] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.908538] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.910879] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.400642] Initialise system trusted keyrings [ 2.401676] Key type blacklist registered [ 2.402862] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.409790] zbud: loaded [ 2.411879] *** VALIDATE nfs *** [ 2.412534] *** VALIDATE nfs4 *** [ 2.413441] pstore: using deflate compression [ 2.416531] Platform Keyring initialized [ 2.494585] NET: Registered protocol family 38 [ 2.496893] Key type asymmetric registered [ 2.498630] Asymmetric key parser 'x509' registered [ 2.500726] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.504143] io scheduler mq-deadline registered [ 2.506063] io scheduler kyber registered [ 2.507913] io scheduler bfq registered [ 2.509787] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.512447] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.515142] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.518099] ACPI: Power Button [PWRF] [ 2.615463] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.700707] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.776867] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.803167] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.831682] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.836702] Non-volatile memory driver v1.3 [ 2.838317] Linux agpgart interface v0.103 [ 2.867418] virtio_blk virtio1: [vda] 133544 512-byte logical blocks (68.4 MB/65.2 MiB) [ 2.870469] vda: detected capacity change from 0 to 68374528 [ 2.897146] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.899428] vdb: detected capacity change from 0 to 1073741824 [ 2.909970] libphy: Fixed MDIO Bus: probed [ 2.915406] usbcore: registered new interface driver usbserial_generic [ 2.917092] usbserial: USB Serial support registered for generic [ 2.919189] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.922594] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.924053] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.926488] mousedev: PS/2 mouse device common for all mice [ 2.928965] rtc_cmos 00:05: RTC can wake from S4 [ 2.931585] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.935516] rtc_cmos 00:05: registered as rtc0 [ 2.937165] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.937996] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.940492] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.942657] intel_pstate: CPU model not supported [ 2.948262] hid: raw HID events driver (C) Jiri Kosina [ 2.949632] usbcore: registered new interface driver usbhid [ 2.951767] usbhid: USB HID core driver [ 2.953434] drop_monitor: Initializing network drop monitor service [ 2.956120] Initializing XFRM netlink socket [ 2.958317] NET: Registered protocol family 10 [ 2.961263] Segment Routing with IPv6 [ 2.963160] NET: Registered protocol family 17 [ 2.965253] mpls_gso: MPLS GSO support [ 2.970966] RAS: Correctable Errors collector initialized. [ 2.973052] AVX version of gcm_enc/dec engaged. [ 2.974598] AES CTR mode by8 optimization enabled [ 3.030437] sched_clock: Marking stable (3030413319, 0)->(3640666540, -610253221) [ 3.033236] registered taskstats version 1 [ 3.034974] Loading compiled-in X.509 certificates [ 3.036422] zswap: loaded using pool lzo/zbud [ 3.067524] Key type big_key registered [ 3.078709] Key type encrypted registered [ 3.080310] ima: No TPM chip found, activating TPM-bypass! [ 3.082054] ima: Allocated hash algorithm: sha1 [ 3.083448] ima: No architecture policies found [ 3.084940] evm: Initialising EVM extended attributes: [ 3.086571] evm: security.selinux [ 3.087614] evm: security.ima [ 3.088564] evm: security.capability [ 3.089684] evm: HMAC attrs: 0x1 [ 3.091966] rtc_cmos 00:05: setting system clock to 2025-08-12 22:56:06 UTC (1755039366) [ 3.098749] debug: unmapping init [mem 0xffffffffb3a03000-0xffffffffb3bfffff] [ 3.101591] debug: unmapping init [mem 0xffffffffb2782000-0xffffffffb2a58fff] [ 3.109073] Write protecting the kernel read-only data: 28672k [ 3.112353] debug: unmapping init [mem 0xffffffffb0e03000-0xffffffffb0ffffff] [ 3.114910] debug: unmapping init [mem 0xffffffffb1714000-0xffffffffb17fffff] [ 3.148954] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.156535] systemd[1]: Detected virtualization kvm. [ 3.158744] systemd[1]: Detected architecture x86-64. [ 3.160914] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.187145] systemd[1]: No hostname configured. [ 3.189123] systemd[1]: Set hostname to . [ 3.191543] random: systemd: uninitialized urandom read (16 bytes read) [ 3.194277] systemd[1]: Initializing machine ID from random generator. [ 3.247167] random: ln: uninitialized urandom read (6 bytes read) [ 3.332135] random: systemd: uninitialized urandom read (16 bytes read) [ 3.333600] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.337551] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.345631] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.980889] device-mapper: uevent: version 1.0.3 [ 3.982226] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.805493] virtio_net virtio0 ens2: renamed from eth0 [ 5.019328] scsi host0: ata_piix [ 5.111390] scsi host1: ata_piix [ 5.124948] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.126928] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.581745] random: fast init done [ 9.598623] random: crng init done [ 9.601722] random: 7 urandom warning(s) missed due to ratelimiting [ 10.204640] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File S[ 11.073293] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) ystems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.683361] printk: systemd: 25 output lines suppressed due to ratelimiting [ 13.035241] SELinux: Disabled at runtime. [ 13.098913] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.104880] systemd[1]: Detected virtualization kvm. [ 13.106346] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.873261] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.877206] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.892565] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.904100] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.915417] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.920930] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.926271] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ 14.025152] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 14.825551] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.444303] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.444444] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.832842] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.869686] EDAC sbridge: Ver: 1.1.2 [ 17.457180] Key type dns_resolver registered [ 17.887320] NFS: Registering the id_resolver key type [ 17.889249] Key type id_resolver registered [ 17.891115] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg452-client login: [ 48.963915] libcfs: loading out-of-tree module taints kernel. [ 48.992263] Key type ._llcrypt registered [ 48.993377] Key type .llcrypt registered [ 49.286572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 49.291808] alg: No test for adler32 (adler32-zlib) [ 50.241638] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 50.518988] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 50.521404] LNet: Accept secure, port 988 [ 52.128148] Key type lgssc registered [ 52.617671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.952728] Lustre: Mounted lustre-client [ 98.332696] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 109.940032] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing check_logdir /tmp/testlogs/ [ 111.376102] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing yml_node [ 112.849269] Lustre: DEBUG MARKER: Client: 2.16.56.30 [ 113.675902] Lustre: DEBUG MARKER: MDS: 2.16.56.30 [ 114.507983] Lustre: DEBUG MARKER: OSS: 2.16.56.30 [ 115.022168] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Aug 12 18:57:57 EDT 2025 [ 116.704399] Lustre: lustre-OST0000-osc-ffff8a5448caf800: disconnect after 20s idle [ 120.135229] LustreError: 7892:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a5448caf800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 120.139355] LustreError: 7892:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 120.151347] Lustre: Unmounted lustre-client [ 129.783202] Key type lgssc unregistered [ 129.928791] LNet: 8337:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 130.982811] LNet: Removed LNI 192.168.204.52@tcp [ 131.276506] Key type .llcrypt unregistered [ 131.277621] Key type ._llcrypt unregistered [ 140.944586] Key type ._llcrypt registered [ 140.945622] Key type .llcrypt registered [ 141.012553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 141.020090] alg: No test for adler32 (adler32-zlib) [ 141.821350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 141.848823] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 141.850408] LNet: Accept secure, port 988 [ 141.857374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 143.163097] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 143.190770] LNet: 9199:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 143.199562] LNet: Removed LNI 192.168.204.52@tcp [ 143.432525] Key type .llcrypt unregistered [ 143.434504] Key type ._llcrypt unregistered [ 145.261898] Key type ._llcrypt registered [ 145.263345] Key type .llcrypt registered [ 145.717164] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:58:28 (1755039508) [ 145.822326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 145.827761] alg: No test for adler32 (adler32-zlib) [ 146.622712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 146.650474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 146.657654] LNet: 9611:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 146.664259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 146.680853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 146.686857] LNet: 9618:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 148.993847] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:58:31 (1755039511) [ 149.025494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 149.037085] LNet: 10189:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 149.049742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 149.082936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 151.441364] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:58:34 (1755039514) [ 151.495378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 151.505741] LNet: 10779:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 151.515714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 151.539449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 153.703410] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:58:36 (1755039516) [ 153.729603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 153.738127] LNet: 11356:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 153.747319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 153.776228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 155.865806] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:58:38 (1755039518) [ 155.890750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 155.905911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 155.949286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 156.081043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 158.277503] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:58:41 (1755039521) [ 158.314779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 158.324360] LNet: 12530:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 158.326351] LNet: 12530:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 158.336349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 158.369469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 158.524511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 158.656926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 160.960473] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:58:43 (1755039523) [ 160.992461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 161.014651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 161.054652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 163.389195] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 18:58:46 (1755039526) [ 163.428933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 163.452437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 163.483185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 163.498582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 163.640975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 163.655350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 163.804067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 163.821831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 163.960068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 163.973771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 164.103974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 164.116537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 164.271568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 164.287776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 164.417943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 164.431957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 164.560748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 164.574973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 164.691691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 164.703434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 164.844161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 164.857622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 164.979080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 164.989992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 165.110846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 165.125954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 165.267276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 165.280913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 167.622808] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 18:58:50 (1755039530) [ 167.663754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 167.674898] LNet: 14392:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 167.678061] LNet: 14392:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 167.694781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 167.730312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 167.744578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 170.112943] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 18:58:53 (1755039533) [ 170.148685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.168073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.208369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 170.226817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 172.474084] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 18:58:55 (1755039535) [ 172.519299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 172.538213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 172.563142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 172.575565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 174.789868] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 18:58:57 (1755039537) [ 174.817121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 174.836275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 174.875203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 174.888312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 177.093329] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 18:59:00 (1755039540) [ 177.133651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 177.154671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 177.184208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 177.197579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 179.509335] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 18:59:02 (1755039542) [ 179.559327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 179.578835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 179.607261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 179.619720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 181.775860] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 18:59:04 (1755039544) [ 181.805748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.824449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.864294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 181.983638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 184.212952] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 18:59:07 (1755039547) [ 184.245848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.258779] LNet: 18496:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 184.262479] LNet: 18496:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 184.274587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.348786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 184.441558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 186.710872] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 18:59:09 (1755039549) [ 186.757145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 186.777725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.823932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 186.905283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 189.157232] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 18:59:12 (1755039552) [ 189.196313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.214716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.367317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 189.459469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 191.824244] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 18:59:14 (1755039554) [ 191.932494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.947641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.980585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 192.004560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 194.546729] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 18:59:17 (1755039557) [ 194.588854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 194.610057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 194.651817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 194.674420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 197.298043] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 18:59:20 (1755039560) [ 197.344746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 197.363720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.387633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 199.877260] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 18:59:22 (1755039562) [ 199.926524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 199.952690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.990601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 202.619897] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 18:59:25 (1755039565) [ 202.671375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.693199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.723394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 202.872721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 205.739552] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 18:59:28 (1755039568) [ 205.787143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.807870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.837204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 205.849806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 208.528094] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 18:59:31 (1755039571) [ 208.559558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 208.575988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 208.616173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 208.630523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 211.072200] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 18:59:33 (1755039573) [ 211.125435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.144310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.190500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 211.209668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 214.113253] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 18:59:36 (1755039576) [ 214.172399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.199302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.253683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 214.275181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 216.842671] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 18:59:39 (1755039579) [ 216.895126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.906770] LNet: 25684:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 216.908790] LNet: 25684:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 216.919704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 216.939902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 216.961807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 216.970614] LNetError: 25697:0:(peer.c:2185:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 216.988949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 217.001926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 219.129876] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 18:59:42 (1755039582) [ 219.157602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 219.173821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 219.205854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BYXR1wpw8x [ 219.231224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BYXR1wpw8x [ 219.251209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BYXR1wpw8x [ 221.464987] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 18:59:44 (1755039584) [ 221.508629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 221.528840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 221.548197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 221.559236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 221.574670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 223.564298] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 18:59:46 (1755039586) [ 223.613573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.636241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.658590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 223.669610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 223.679733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 223.690391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 223.705582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 223.715475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 223.726131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 223.740387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 223.757106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 223.769261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 223.782119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 223.792653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 223.803869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 223.817169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 223.831530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 223.848846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 223.862402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 223.879882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 223.894372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 223.903088] LNetError: 27496:0:(router.c:632:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 226.001951] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 18:59:48 (1755039588) [ 226.041941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.062471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.096991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 228.458764] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 18:59:51 (1755039591) [ 229.195582] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg452-server [ 229.930961] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 18:59:52 (1755039592) [ 229.963604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.982831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.011265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 230.024092] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 230.025627] LNet: Accept secure, port 988 [ 230.603372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[53-57]@tcp [ 230.611284] LNet: 29207:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 230.618823] LNetError: 28999:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 230.623128] LNetError: 28999:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.56@tcp has gone from up to down [ 230.626591] LNetError: 29210:0:(router.c:334:lnet_is_route_alive()) peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 232.884275] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 18:59:55 (1755039595) [ 232.906353] LNetError: 29791:0:(router.c:334:lnet_is_route_alive()) peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 233.673927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg452-server [ 233.696508] LNetError: 29006:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.55@tcp has gone from up to down [ 234.255388] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 18:59:57 (1755039597) [ 234.287885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.367416] LNet: Removed LNI 192.168.204.52@tcp [ 235.381897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.416600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 235.433412] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 235.437285] LNet: Accept secure, port 988 [ 235.784462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[53-58/2]@tcp [ 235.796432] LNet: 30255:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 235.800949] LNet: 30255:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 235.811492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[53-58/2]@tcp [ 238.434723] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:00:01 (1755039601) [ 238.501453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.518896] LNet: Removed LNI 192.168.204.52@tcp [ 238.526701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.558810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 238.581674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.593710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.614994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 238.634414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.650632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.676300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 238.705933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.719593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.742420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 238.773660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.788474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.809723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 238.843133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.856813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.879622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 238.917637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.937546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.970129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 239.018536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.039425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.073032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 239.110481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.125283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.148370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 239.186426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.204175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.233610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 239.267497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.280047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.299574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 239.320897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.335125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.358644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 242.109431] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:00:04 (1755039604) [ 242.794592] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg452-server [ 243.371576] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:00:06 (1755039606) [ 244.020720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg452-server [ 244.732524] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:00:07 (1755039607) [ 245.058629] Key type .llcrypt unregistered [ 245.060498] Key type ._llcrypt unregistered [ 247.279837] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 247.335173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 247.337474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 247.338947] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 247.480520] Key type ._llcrypt registered [ 247.481482] Key type .llcrypt registered [ 247.549530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 247.556848] alg: No test for adler32 (adler32-zlib) [ 248.365145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 248.387565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 248.397383] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 248.399244] LNet: Accept secure, port 988 [ 248.635934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 248.644380] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 249.405306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 249.415240] LNet: Removed LNI 10.1.2.3@tcp [ 249.463150] LNet: 32932:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 249.472534] LNet: Removed LNI 192.168.204.52@tcp [ 249.728445] Key type .llcrypt unregistered [ 249.729725] Key type ._llcrypt unregistered [ 253.389579] Key type ._llcrypt registered [ 253.390862] Key type .llcrypt registered [ 253.968558] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:00:16 (1755039616) [ 254.170369] Key type .llcrypt unregistered [ 254.171662] Key type ._llcrypt unregistered [ 256.078261] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 256.133901] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 256.136716] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 256.139561] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 256.284889] Key type ._llcrypt registered [ 256.286753] Key type .llcrypt registered [ 256.349915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 256.356268] alg: No test for adler32 (adler32-zlib) [ 257.165422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 257.187876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 257.197299] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 257.200340] LNet: Accept secure, port 988 [ 257.460720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 257.468794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 258.297360] LNet: 34387:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.300070] LNet: Removed LNI 192.168.204.52@tcp [ 258.307329] LNet: Removed LNI 10.1.2.3@tcp [ 258.542752] Key type .llcrypt unregistered [ 258.543801] Key type ._llcrypt unregistered [ 262.268353] Key type ._llcrypt registered [ 262.269401] Key type .llcrypt registered [ 262.736584] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:00:25 (1755039625) [ 262.917308] Key type .llcrypt unregistered [ 262.918352] Key type ._llcrypt unregistered [ 264.936226] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 264.976351] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 264.978295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 264.979815] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 265.115456] Key type ._llcrypt registered [ 265.116961] Key type .llcrypt registered [ 265.180048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 265.185676] alg: No test for adler32 (adler32-zlib) [ 265.991976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.013542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 266.023286] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 266.024809] LNet: Accept secure, port 988 [ 266.729574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 266.736567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 267.486225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 267.493592] LNet: Removed LNI 10.1.2.3@tcp [ 267.499737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 267.512611] LNet: Removed LNI 10.1.2.31@tcp [ 267.518754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 267.528291] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 267.794403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 267.804332] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 268.157483] LNet: 36262:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 268.159919] LNet: Removed LNI 10.1.2.3@tcp [ 268.168559] LNet: Removed LNI 10.1.2.31@tcp [ 268.436611] Key type .llcrypt unregistered [ 268.437867] Key type ._llcrypt unregistered [ 272.082724] Key type ._llcrypt registered [ 272.083920] Key type .llcrypt registered [ 272.613355] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:00:35 (1755039635) [ 272.758797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 272.764800] alg: No test for adler32 (adler32-zlib) [ 273.569516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.589267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 273.598432] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 273.600482] LNet: Accept secure, port 988 [ 273.939127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.947942] LNet: 37206:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 273.959530] LNet: Removed LNI 192.168.204.52@tcp [ 273.971329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.004731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.52@tcp [ 274.020179] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 274.023549] LNet: Accept secure, port 988 [ 274.415307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.52@tcp [ 274.433634] LNet: Removed LNI 192.168.204.52@tcp [ 274.464189] LNet: 37338:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 274.730413] Key type .llcrypt unregistered [ 274.731713] Key type ._llcrypt unregistered [ 278.700717] Key type ._llcrypt registered [ 278.701934] Key type .llcrypt registered [ 279.120800] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:00:42 (1755039642) [ 279.750768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg452-server [ 280.355695] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:00:43 (1755039643) [ 280.607372] Key type .llcrypt unregistered [ 280.609430] Key type ._llcrypt unregistered [ 282.947801] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 282.978602] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 282.980612] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 282.982174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 283.100610] Key type ._llcrypt registered [ 283.101480] Key type .llcrypt registered [ 283.165196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 283.171071] alg: No test for adler32 (adler32-zlib) [ 283.979516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.000750] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 284.002492] LNet: Accept secure, port 988 [ 284.305342] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 285.250301] LNet: 39134:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.252932] LNet: Removed LNI 192.168.204.52@tcp1 [ 285.260456] LNet: Removed LNI 10.1.2.3@tcp2 [ 285.503471] Key type .llcrypt unregistered [ 285.504694] Key type ._llcrypt unregistered [ 289.171559] Key type ._llcrypt registered [ 289.172841] Key type .llcrypt registered [ 289.693586] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:00:52 (1755039652) [ 289.827129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 289.833261] alg: No test for adler32 (adler32-zlib) [ 290.636888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.658391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 290.666302] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 290.668344] LNet: Accept secure, port 988 [ 290.907825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.52 [ 290.915813] LNet: Removed LNI 192.168.204.52@tcp [ 292.920582] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:00:55 (1755039655) [ 292.949630] LNet: 40653:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.959549] LNet: Removed LNI 192.168.204.52@tcp [ 293.179334] Key type .llcrypt unregistered [ 293.180497] Key type ._llcrypt unregistered [ 295.088274] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 295.199691] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 295.201167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 295.202767] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 295.326060] Key type ._llcrypt registered [ 295.327103] Key type .llcrypt registered [ 295.386123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 295.392359] alg: No test for adler32 (adler32-zlib) [ 296.228869] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 296.230874] LNet: Accept secure, port 988 [ 298.786729] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:01:01 (1755039661) [ 298.819180] LNet: 41530:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.826878] LNet: Removed LNI 10.1.2.3@tcp [ 299.065712] Key type .llcrypt unregistered [ 299.066898] Key type ._llcrypt unregistered [ 301.113631] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 301.225888] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 301.228276] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 301.230659] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 301.394828] Key type ._llcrypt registered [ 301.397421] Key type .llcrypt registered [ 301.467346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 301.472680] alg: No test for adler32 (adler32-zlib) [ 302.326051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 302.328694] LNet: Accept secure, port 988 [ 304.645529] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:01:07 (1755039667) [ 304.682571] LNet: 42393:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.690800] LNet: Removed LNI 10.1.2.3@tcp [ 304.903495] Key type .llcrypt unregistered [ 304.904664] Key type ._llcrypt unregistered [ 306.976534] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 307.094220] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 307.096470] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 307.098266] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 307.220322] Key type ._llcrypt registered [ 307.221703] Key type .llcrypt registered [ 307.286807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 307.291695] alg: No test for adler32 (adler32-zlib) [ 308.109881] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 308.112143] LNet: Accept secure, port 988 [ 310.851471] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:01:13 (1755039673) [ 310.907208] LNet: 43258:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.918612] LNet: Removed LNI 10.1.2.3@tcp [ 311.270797] Key type .llcrypt unregistered [ 311.272466] Key type ._llcrypt unregistered [ 313.454721] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 313.552627] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 313.553834] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 313.555153] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 313.656925] Key type ._llcrypt registered [ 313.657762] Key type .llcrypt registered [ 313.708688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 313.713368] alg: No test for adler32 (adler32-zlib) [ 314.537970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.572513] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 314.574288] LNet: Accept secure, port 988 [ 317.713254] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:01:20 (1755039680) [ 317.871747] LNet: 44463:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.883755] LNet: Removed LNI 10.1.2.3@tcp [ 318.142395] Key type .llcrypt unregistered [ 318.143694] Key type ._llcrypt unregistered [ 320.125609] Key type ._llcrypt registered [ 320.126840] Key type .llcrypt registered [ 320.186462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 320.191683] alg: No test for adler32 (adler32-zlib) [ 320.994044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 321.003282] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 321.004773] LNet: Accept secure, port 988 [ 324.218212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 324.414922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_interrupt [ 324.429189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_interrupt [ 324.441746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 324.449102] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 324.523642] LNet: 44875:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 324.547637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 324.558365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 324.569391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 324.767365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_dropped [ 324.783722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_dropped [ 324.803335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 324.942400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 324.956304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 324.968809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 325.183104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_aborted [ 325.199374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_aborted [ 325.215110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 325.223433] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 325.226013] LNet: Skipped 1 previous similar message [ 325.308763] LNet: 45045:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 325.311066] LNet: 45045:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 325.333323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 325.343788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 325.354471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 325.512875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_no_route [ 325.525254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_no_route [ 325.543565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 325.658609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 325.676329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 325.694829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 325.860570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_timeout [ 325.872871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_timeout [ 325.887875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 325.992863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 326.003707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 326.015367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 326.159922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_error [ 326.173376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_error [ 326.189874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 326.290935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 326.301301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 326.311181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 326.396714] LNet: 45336:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.462656] LNet: Removed LNI 192.168.204.52@tcp [ 327.655507] Key type .llcrypt unregistered [ 327.656922] Key type ._llcrypt unregistered [ 333.522709] Key type ._llcrypt registered [ 333.523655] Key type .llcrypt registered [ 334.129452] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:01:36 (1755039696) [ 334.555227] Key type .llcrypt unregistered [ 334.556279] Key type ._llcrypt unregistered [ 336.487200] Key type ._llcrypt registered [ 336.489095] Key type .llcrypt registered [ 336.553352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 336.559280] alg: No test for adler32 (adler32-zlib) [ 337.372441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 337.383176] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 337.384914] LNet: Accept secure, port 988 [ 340.340686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 343.404464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.414783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 343.423854] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 343.853722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_interrupt [ 343.872957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -r 1 -e local_interrupt [ 343.890511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_interrupt [ 343.902618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -r 1 -e local_interrupt [ 343.915832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -r 1 -e local_interrupt [ 343.927029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -r 1 -e local_interrupt [ 343.938399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -r 1 -e local_interrupt [ 343.950823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -r 1 -e local_interrupt [ 343.966400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 343.973495] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 343.977976] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 344.068266] LNet: 46770:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 344.092521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 344.103512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 344.114317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 344.264981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_dropped [ 344.275835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -r 1 -e local_dropped [ 344.286507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_dropped [ 344.296924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -r 1 -e local_dropped [ 344.310000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -r 1 -e local_dropped [ 344.321720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -r 1 -e local_dropped [ 344.334605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -r 1 -e local_dropped [ 344.345374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -r 1 -e local_dropped [ 344.357099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 344.467303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 344.479040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 344.491826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 344.656530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_aborted [ 344.674733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -r 1 -e local_aborted [ 344.693976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_aborted [ 344.709520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -r 1 -e local_aborted [ 344.725202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -r 1 -e local_aborted [ 344.739120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -r 1 -e local_aborted [ 344.752655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -r 1 -e local_aborted [ 344.766027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -r 1 -e local_aborted [ 344.779177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 344.788074] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 344.791124] LNet: Skipped 6 previous similar messages [ 344.868909] LNet: 46964:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 344.870949] LNet: 46964:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 344.893026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 344.903593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 344.914640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 345.055217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_no_route [ 345.070055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -r 1 -e local_no_route [ 345.084281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_no_route [ 345.097954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -r 1 -e local_no_route [ 345.112371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -r 1 -e local_no_route [ 345.128522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -r 1 -e local_no_route [ 345.145414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -r 1 -e local_no_route [ 345.157319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -r 1 -e local_no_route [ 345.168541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 345.267327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 345.276850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 345.286279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 345.425627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_timeout [ 345.437107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -r 1 -e local_timeout [ 345.447640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_timeout [ 345.458645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -r 1 -e local_timeout [ 345.469022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -r 1 -e local_timeout [ 345.480565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -r 1 -e local_timeout [ 345.492535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -r 1 -e local_timeout [ 345.504117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -r 1 -e local_timeout [ 345.516040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 345.622578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 345.634395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 345.646092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 345.790611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e local_error [ 345.803905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -r 1 -e local_error [ 345.817163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e local_error [ 345.831581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -r 1 -e local_error [ 345.843904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -r 1 -e local_error [ 345.854704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -r 1 -e local_error [ 345.868065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -r 1 -e local_error [ 345.882674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -r 1 -e local_error [ 345.895457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 345.905696] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 345.908468] LNet: Skipped 8 previous similar messages [ 345.980058] LNet: 47255:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 345.982240] LNet: 47255:0:(api-ni.c:337:recovery_interval_set()) Skipped 2 previous similar messages [ 346.005138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 346.016133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 346.025552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 346.971819] LNet: 47325:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.000114] LNet: Removed LNI 192.168.204.52@tcp [ 349.030399] LNet: Removed LNI 192.168.204.52@tcp1 [ 349.179490] Key type .llcrypt unregistered [ 349.180411] Key type ._llcrypt unregistered [ 354.109115] Key type ._llcrypt registered [ 354.110072] Key type .llcrypt registered [ 354.514968] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:01:57 (1755039717) [ 354.749329] Key type .llcrypt unregistered [ 354.750261] Key type ._llcrypt unregistered [ 356.485167] Key type ._llcrypt registered [ 356.486160] Key type .llcrypt registered [ 356.538876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 356.543962] alg: No test for adler32 (adler32-zlib) [ 357.342821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 357.354458] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 357.355870] LNet: Accept secure, port 988 [ 360.085870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 360.252563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e remote_dropped [ 360.263208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e remote_dropped [ 360.274285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 360.280701] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 360.358574] LNet: 48545:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 360.379755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 360.390748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 360.401437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 360.546758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e remote_error [ 360.557838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e remote_error [ 360.569911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 360.677865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 360.689757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 360.701291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 360.858120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e remote_timeout [ 360.869525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e remote_timeout [ 360.880843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 360.887408] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 360.889133] LNet: Skipped 1 previous similar message [ 360.969596] LNet: 48716:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 360.971758] LNet: 48716:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 360.992952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 361.003149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 361.013423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 361.102710] LNet: 48752:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.116416] LNet: Removed LNI 192.168.204.52@tcp [ 361.258766] Key type .llcrypt unregistered [ 361.259743] Key type ._llcrypt unregistered [ 366.406353] Key type ._llcrypt registered [ 366.408356] Key type .llcrypt registered [ 366.938521] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:02:09 (1755039729) [ 367.285534] Key type .llcrypt unregistered [ 367.286649] Key type ._llcrypt unregistered [ 369.322071] Key type ._llcrypt registered [ 369.323016] Key type .llcrypt registered [ 369.378862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 369.383898] alg: No test for adler32 (adler32-zlib) [ 370.190247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 370.201977] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 370.204212] LNet: Accept secure, port 988 [ 372.766134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 375.462319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.477181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 375.486388] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 375.949838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e remote_dropped [ 375.963984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -r 1 -e remote_dropped [ 375.978768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e remote_dropped [ 375.995256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -r 1 -e remote_dropped [ 376.009046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -r 1 -e remote_dropped [ 376.023155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -r 1 -e remote_dropped [ 376.037505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -r 1 -e remote_dropped [ 376.052106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -r 1 -e remote_dropped [ 376.072834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 376.083557] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 376.087655] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 376.191341] LNet: 50185:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 376.219459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 376.234850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 376.249829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 376.265282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 376.435797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e remote_error [ 376.450176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -r 1 -e remote_error [ 376.465395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e remote_error [ 376.482904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -r 1 -e remote_error [ 376.499748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -r 1 -e remote_error [ 376.515481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -r 1 -e remote_error [ 376.530199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -r 1 -e remote_error [ 376.541407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -r 1 -e remote_error [ 376.552502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 376.674313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 376.686627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 376.697944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 376.709244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 376.871370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e remote_timeout [ 376.884258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -r 1 -e remote_timeout [ 376.896637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e remote_timeout [ 376.909660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -r 1 -e remote_timeout [ 376.921292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -r 1 -e remote_timeout [ 376.933738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -r 1 -e remote_timeout [ 376.944905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -r 1 -e remote_timeout [ 376.955789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -r 1 -e remote_timeout [ 376.966801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 376.972735] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 376.974562] LNet: Skipped 3 previous similar messages [ 377.063608] LNet: 50383:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 377.065652] LNet: 50383:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 377.089255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 377.100636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 377.113070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 377.124792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 379.173628] LNet: 50455:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.192167] LNet: Removed LNI 192.168.204.52@tcp [ 380.199330] LNet: Removed LNI 192.168.204.52@tcp1 [ 380.332462] Key type .llcrypt unregistered [ 380.333682] Key type ._llcrypt unregistered [ 385.144534] Key type ._llcrypt registered [ 385.145543] Key type .llcrypt registered [ 385.597632] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:02:28 (1755039748) [ 385.789396] Key type .llcrypt unregistered [ 385.790390] Key type ._llcrypt unregistered [ 387.830059] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 387.860568] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 387.863216] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 387.867391] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 387.998823] Key type ._llcrypt registered [ 387.999804] Key type .llcrypt registered [ 388.048269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 388.054563] alg: No test for adler32 (adler32-zlib) [ 388.870946] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 388.872322] LNet: Accept secure, port 988 [ 388.889574] LNet: 51517:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.905480] LNet: Removed LNI 192.168.204.52@tcp [ 389.180303] Key type .llcrypt unregistered [ 389.181246] Key type ._llcrypt unregistered [ 391.070375] Key type ._llcrypt registered [ 391.071747] Key type .llcrypt registered [ 391.129531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 391.135072] alg: No test for adler32 (adler32-zlib) [ 391.947441] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 391.948873] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 391.950201] LNet: Accept secure, port 988 [ 391.961962] LNet: 51786:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.965501] LNet: Removed LNI 192.168.204.52@tcp [ 391.975476] LNet: Removed LNI 10.1.2.3@tcp1 [ 392.236462] Key type .llcrypt unregistered [ 392.238480] Key type ._llcrypt unregistered [ 394.407503] Key type ._llcrypt registered [ 394.409377] Key type .llcrypt registered [ 394.470465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 394.476553] alg: No test for adler32 (adler32-zlib) [ 395.284619] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 395.286220] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 395.287711] LNet: Accept secure, port 988 [ 395.303369] LNet: 52055:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.306520] LNet: Removed LNI 192.168.204.52@tcp [ 395.319553] LNet: Removed LNI 10.1.2.3@tcp [ 395.597424] Key type .llcrypt unregistered [ 395.598701] Key type ._llcrypt unregistered [ 397.746359] Key type ._llcrypt registered [ 397.748015] Key type .llcrypt registered [ 397.806850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 397.812457] alg: No test for adler32 (adler32-zlib) [ 398.624869] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 398.626390] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 398.627694] LNet: Accept secure, port 988 [ 398.638410] LNet: 52324:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.640915] LNet: Removed LNI 192.168.204.52@tcp [ 398.654504] LNet: Removed LNI 10.1.2.3@tcp [ 398.887320] Key type .llcrypt unregistered [ 398.888402] Key type ._llcrypt unregistered [ 401.030589] Key type ._llcrypt registered [ 401.031763] Key type .llcrypt registered [ 401.083096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 401.088191] alg: No test for adler32 (adler32-zlib) [ 401.889867] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 401.891563] LNetError: 52584:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 401.893771] LNetError: Error -100 starting up LNI tcp [ 401.902286] LNet: Removed LNI 192.168.204.52@tcp [ 402.138555] Key type .llcrypt unregistered [ 402.139967] Key type ._llcrypt unregistered [ 405.937497] Key type ._llcrypt registered [ 405.938642] Key type .llcrypt registered [ 406.417014] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:02:49 (1755039769) [ 406.740552] Key type .llcrypt unregistered [ 406.741575] Key type ._llcrypt unregistered [ 408.669613] Key type ._llcrypt registered [ 408.670739] Key type .llcrypt registered [ 408.730245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 408.736649] alg: No test for adler32 (adler32-zlib) [ 409.537695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 409.550949] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 409.554032] LNet: Accept secure, port 988 [ 412.258227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 412.489584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e network_timeout [ 412.504474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e network_timeout [ 412.516558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 412.524639] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 412.614576] LNet: 53810:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 412.636955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 412.648278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 412.738668] LNet: 53844:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.747491] LNet: Removed LNI 192.168.204.52@tcp [ 412.915372] Key type .llcrypt unregistered [ 412.916319] Key type ._llcrypt unregistered [ 418.611388] Key type ._llcrypt registered [ 418.612464] Key type .llcrypt registered [ 418.676770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 418.681721] alg: No test for adler32 (adler32-zlib) [ 419.491257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 419.500753] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 419.502164] LNet: Accept secure, port 988 [ 422.476614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 425.197532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.212560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 425.221924] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 425.689976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -r 1 -e network_timeout [ 425.704296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -r 1 -e network_timeout [ 425.715894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -r 1 -e network_timeout [ 425.728938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -r 1 -e network_timeout [ 425.741436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -r 1 -e network_timeout [ 425.753980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -r 1 -e network_timeout [ 425.766214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -r 1 -e network_timeout [ 425.777498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -r 1 -e network_timeout [ 425.789797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 425.796093] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 425.899430] LNet: 54691:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 425.927739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 425.938731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 428.101771] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 428.193898] LNet: 54760:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.197135] LNet: Removed LNI 192.168.204.52@tcp [ 428.206409] LNet: Removed LNI 192.168.204.52@tcp1 [ 428.356380] Key type .llcrypt unregistered [ 428.357330] Key type ._llcrypt unregistered [ 433.804964] Key type ._llcrypt registered [ 433.805983] Key type .llcrypt registered [ 434.337176] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:03:17 (1755039797) [ 434.447968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 434.452715] alg: No test for adler32 (adler32-zlib) [ 435.253782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.271366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 435.278620] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 435.279923] LNet: Accept secure, port 988 [ 435.478294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 435.484387] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 435.717199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 435.738745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 435.752597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 435.762730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 435.773341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 435.783332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 435.794222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 435.800117] LNetError: 55848:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 436.320274] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 436.320300] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 438.368756] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 438.947478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 442.464771] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 443.124535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 443.134923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 443.152737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.159342] LNet: 55966:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.161564] LNet: Removed LNI 192.168.204.52@tcp [ 443.166249] LNet: Removed LNI 192.168.204.52@tcp1 [ 443.172165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.188617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 443.196303] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 443.198129] LNet: Accept secure, port 988 [ 443.402478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 443.410699] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 443.664728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 443.679986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 443.701267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 443.714978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 443.726075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 443.738605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 443.752546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 443.767476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 443.777975] LNetError: 56232:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 446.240963] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 446.244828] LNet: Skipped 1 previous similar message [ 446.931616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 450.336746] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 455.217483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 455.230978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 455.246106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 457.463620] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:03:40 (1755039820) [ 457.494587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.504881] LNet: 56952:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.528097] LNet: Removed LNI 192.168.204.52@tcp [ 458.536104] LNet: Removed LNI 192.168.204.52@tcp1 [ 458.546194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.565693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 458.574232] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 458.576045] LNet: Accept secure, port 988 [ 458.801918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 458.808802] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 459.104557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 459.138822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 459.155771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 459.175646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 459.193956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 459.209458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 459.222635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 0 [ 459.271289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 459.616222] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.52@tcp [ 460.640358] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 460.643326] LNet: Skipped 1 previous similar message [ 475.767326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 475.813262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 475.828393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 475.842671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 500 [ 475.875908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 483.089642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 483.111907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.119750] LNet: 57468:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 484.128151] LNet: Removed LNI 192.168.204.52@tcp [ 484.135392] LNet: Removed LNI 192.168.204.52@tcp1 [ 484.143880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.165830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 484.173473] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 484.174870] LNet: Accept secure, port 988 [ 484.428077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 484.438722] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 484.737713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 484.751598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 484.771530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 484.783147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 484.794091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 484.804421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 484.817614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 484.833932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 0 [ 485.217279] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 485.219070] LNet: Skipped 3 previous similar messages [ 485.933649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 496.322138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 496.337578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 496.352192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 499.054546] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:04:21 (1755039861) [ 499.215670] LNet: 58517:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.256169] LNet: Removed LNI 192.168.204.52@tcp [ 500.264807] LNet: Removed LNI 192.168.204.52@tcp1 [ 500.514645] Key type .llcrypt unregistered [ 500.515652] Key type ._llcrypt unregistered [ 502.556729] Key type ._llcrypt registered [ 502.558188] Key type .llcrypt registered [ 502.622469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 502.627330] alg: No test for adler32 (adler32-zlib) [ 503.432883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 503.444253] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 503.445738] LNet: Accept secure, port 988 [ 506.851898] Lustre: DEBUG MARKER: Initial discovery [ 506.890905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 507.833780] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 507.876704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 507.896686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 507.907337] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 507.915994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 507.928833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.152@tcp -r 1 -m GET -l 3 [ 507.942839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.152@tcp -r 1 -m GET -e local_error [ 507.956210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.152@tcp -r 1 -m PUT -l 6 [ 508.447491] Lustre: DEBUG MARKER: Force oleg452-client.virtnet to discover 192.168.204.152@tcp (in background) [ 508.474596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 508.980980] Lustre: DEBUG MARKER: Force oleg452-server to discover 192.168.204.52@tcp [ 509.753640] Lustre: DEBUG MARKER: Wait for 59266 [ 511.520443] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 512.211066] Lustre: DEBUG MARKER: Finished wait on 59266 [ 512.231648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 512.255516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 512.364691] LNet: 59721:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.249827] LNetError: 58771:0:(socklnd.c:1267:ksocknal_create_conn()) Not creating conn 12345-192.168.204.152@tcp type 1: peer_ni/conn_cb removed [ 513.255360] LNetError: Unexpected error -116 connecting to 192.168.204.152@tcp at host 192.168.204.152:988 [ 513.382668] LNet: Removed LNI 192.168.204.52@tcp [ 513.604521] Key type .llcrypt unregistered [ 513.605844] Key type ._llcrypt unregistered [ 519.078139] Key type ._llcrypt registered [ 519.079420] Key type .llcrypt registered [ 519.578372] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:04:42 (1755039882) [ 519.832253] Key type .llcrypt unregistered [ 519.834191] Key type ._llcrypt unregistered [ 521.733986] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 521.805053] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 521.806759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 521.808265] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 521.945278] Key type ._llcrypt registered [ 521.946200] Key type .llcrypt registered [ 522.005988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 522.010733] alg: No test for adler32 (adler32-zlib) [ 522.821132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.844799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 522.853762] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 522.855335] LNet: Accept secure, port 988 [ 523.111968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 523.125149] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 526.197637] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:04:49 (1755039889) [ 526.228652] LNet: 61751:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.233795] LNet: Removed LNI 192.168.204.52@tcp [ 526.240865] LNet: Removed LNI 10.1.2.3@tcp [ 526.509380] Key type .llcrypt unregistered [ 526.510587] Key type ._llcrypt unregistered [ 528.432038] Key type ._llcrypt registered [ 528.433133] Key type .llcrypt registered [ 528.490125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 528.496040] alg: No test for adler32 (adler32-zlib) [ 529.299947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.323723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 529.336456] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 529.339526] LNet: Accept secure, port 988 [ 529.597691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 529.606741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 529.983807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.52@tcp [ 531.040327] LNetError: Unexpected error -22 connecting to 192.168.204.52@tcp at host 192.168.204.52:988 [ 532.177506] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:04:55 (1755039895) [ 532.207500] LNet: 62865:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.216140] LNet: Removed LNI 192.168.204.52@tcp [ 533.223536] LNet: Removed LNI 10.1.2.3@tcp [ 533.439320] Key type .llcrypt unregistered [ 533.440976] Key type ._llcrypt unregistered [ 535.339661] Key type ._llcrypt registered [ 535.342649] Key type .llcrypt registered [ 535.422321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 535.427434] alg: No test for adler32 (adler32-zlib) [ 536.238359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.262080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 536.275656] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 536.278337] LNet: Accept secure, port 988 [ 536.552810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 536.560046] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 536.952648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.52@tcp1 --nid 192.168.204.52@tcp2 [ 538.501975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp1 192.168.204.52@tcp2 [ 538.509429] LNetError: 63696:0:(lib-move.c:2225:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.52@tcp1 to 192.168.204.52@tcp2 - Route table may be misconfigured [ 538.517722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp2 192.168.204.52@tcp1 [ 540.792801] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:05:03 (1755039903) [ 540.834506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.847794] LNet: 64264:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.856151] LNet: Removed LNI 192.168.204.52@tcp1 [ 542.886524] LNet: Removed LNI 192.168.204.52@tcp2 [ 542.893870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.911073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 542.918526] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 542.920125] LNet: Accept secure, port 988 [ 543.135356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 543.144439] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 543.429387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 543.446451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.52@tcp [ 543.461324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp1 192.168.204.52@tcp1 [ 543.475765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.52@tcp -d 192.168.204.52@tcp -e network_timeout [ 543.487393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -e network_timeout [ 543.498754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -e network_timeout [ 543.509837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -e network_timeout [ 543.522595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 543.528836] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 543.535273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 543.569163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 543.604293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 543.904083] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 545.849404] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:05:08 (1755039908) [ 545.876921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.884834] LNet: 65119:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.912132] LNet: Removed LNI 192.168.204.52@tcp [ 547.942885] LNet: Removed LNI 192.168.204.52@tcp1 [ 547.954887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.988845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 548.097726] LNet: 65160:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.274651] Key type .llcrypt unregistered [ 548.275648] Key type ._llcrypt unregistered [ 552.052831] Key type ._llcrypt registered [ 552.054271] Key type .llcrypt registered [ 552.505624] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:05:15 (1755039915) [ 552.625912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 552.631686] alg: No test for adler32 (adler32-zlib) [ 553.434676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.935109] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 554.548957] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:05:17 (1755039917) [ 554.580710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.590584] LNet: 66247:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.600642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.630470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 554.640357] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 554.642535] LNet: Accept secure, port 988 [ 554.896734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 554.903729] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 555.244956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 555.260037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp1 [ 555.274182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp1 [ 557.444167] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:05:20 (1755039920) [ 558.077261] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg452-server [ 558.655175] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:05:21 (1755039921) [ 559.300451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg452-server [ 559.886165] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:05:22 (1755039922) [ 560.520709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg452-server [ 561.096988] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:05:24 (1755039924) [ 561.644842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg452-server [ 562.218804] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:05:25 (1755039925) [ 562.771896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg452-server [ 563.298894] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:05:26 (1755039926) [ 563.837830] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg452-server [ 564.381145] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:05:27 (1755039927) [ 564.998530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg452-server [ 565.561281] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:05:28 (1755039928) [ 566.211962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg452-server [ 566.791747] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:05:29 (1755039929) [ 566.837117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.847111] LNet: 69778:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.904193] LNet: Removed LNI 192.168.204.52@tcp [ 568.934972] LNet: Removed LNI 192.168.204.52@tcp1 [ 568.945176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.963964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 568.972177] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 568.974170] LNet: Accept secure, port 988 [ 569.289467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 569.327971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 569.394876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.402497] LNet: 69925:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.470448] LNet: Removed LNI 192.168.204.52@tcp [ 570.478792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.499841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 570.511040] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 570.512914] LNet: Accept secure, port 988 [ 570.787267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 571.889347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.898985] LNet: 70077:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.966397] LNet: Removed LNI 192.168.204.52@tcp [ 572.976640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.002071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 573.014799] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 573.016273] LNet: Accept secure, port 988 [ 573.252871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 573.279273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 573.337658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.374657] LNet: Removed LNI 192.168.204.52@tcp [ 574.390272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.413197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 574.421552] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 574.422949] LNet: Accept secure, port 988 [ 574.666706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 574.721109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.782600] LNet: Removed LNI 192.168.204.52@tcp [ 575.796515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.828671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 575.844052] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 575.847030] LNet: Accept secure, port 988 [ 576.108906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 576.136622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 576.202469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.212623] LNet: 70507:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.215130] LNet: 70507:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 577.254619] LNet: Removed LNI 192.168.204.52@tcp [ 577.269414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.305485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 577.323170] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 577.326728] LNet: Accept secure, port 988 [ 577.609469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 577.671641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.726417] LNet: Removed LNI 192.168.204.52@tcp [ 578.734777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.751779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 578.760460] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 578.761738] LNet: Accept secure, port 988 [ 578.972674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 579.001377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 579.074744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.134338] LNet: Removed LNI 192.168.204.52@tcp [ 580.142473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.161260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 580.170233] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 580.171670] LNet: Accept secure, port 988 [ 580.529252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 580.629526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.670664] LNet: Removed LNI 192.168.204.52@tcp [ 581.685478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.721428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 581.736791] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 581.741194] LNet: Accept secure, port 988 [ 582.025529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 582.075656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 582.185972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.206307] LNet: Removed LNI 192.168.204.52@tcp [ 583.213509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.229938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 583.237259] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 583.238621] LNet: Accept secure, port 988 [ 583.471784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 584.618253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.627542] LNet: 71231:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.630725] LNet: 71231:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 585.702615] LNet: Removed LNI 192.168.204.52@tcp [ 585.717624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.752859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.768194] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 585.771443] LNet: Accept secure, port 988 [ 586.098869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 586.134838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 586.239118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.302470] LNet: Removed LNI 192.168.204.52@tcp [ 587.310321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.328356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 587.335916] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 587.337789] LNet: Accept secure, port 988 [ 587.589996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 588.705524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.734299] LNet: Removed LNI 192.168.204.52@tcp [ 589.743202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.761301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 589.769157] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 589.770778] LNet: Accept secure, port 988 [ 589.981535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 590.000661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.013315] LNet: Removed LNI 192.168.204.52@tcp [ 590.019861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.036534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 590.043890] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 590.045566] LNet: Accept secure, port 988 [ 590.260549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 592.569818] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:05:55 (1755039955) [ 592.621540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.643128] LNet: Removed LNI 192.168.204.52@tcp [ 592.650734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.671106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 592.680693] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 592.682715] LNet: Accept secure, port 988 [ 593.108726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.127416] LNet: Removed LNI 192.168.204.52@tcp231 [ 593.135430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.161735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 593.170994] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 593.172738] LNet: Accept secure, port 988 [ 593.174102] LNet: 72500:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 593.310241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 593.325407] LNet: Removed LNI 192.168.204.52@tcp231 [ 593.331027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 593.339535] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 593.731833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.748373] LNet: Removed LNI 192.168.204.52@tcp231 [ 593.756025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.780546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 593.790724] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/0] [ 593.792506] LNet: Accept secure, port 988 [ 593.794764] LNet: 72649:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 593.912127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 593.932474] LNet: Removed LNI 192.168.204.52@tcp231 [ 593.939610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 593.948245] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/0] [ 594.349569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.365505] LNet: Removed LNI 192.168.204.52@tcp231 [ 594.374639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.400140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 594.408646] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/60] [ 594.410356] LNet: Accept secure, port 988 [ 594.552104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 594.571846] LNet: Removed LNI 192.168.204.52@tcp231 [ 594.578417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 594.586125] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/60] [ 594.916896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.931288] LNet: Removed LNI 192.168.204.52@tcp231 [ 594.936986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.956776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 594.964685] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 594.966570] LNet: Accept secure, port 988 [ 594.968656] LNet: 72946:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 594.971407] LNet: 72946:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 595.099696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 595.121456] LNet: Removed LNI 192.168.204.52@tcp231 [ 595.127246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 595.135961] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 597.628764] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:06:00 (1755039960) [ 597.678743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.698183] LNet: Removed LNI 192.168.204.52@tcp231 [ 597.709820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.732402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 597.740277] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 597.741796] LNet: Accept secure, port 988 [ 597.994684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 598.008925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 598.024450] LNet: Removed LNI 192.168.204.52@tcp232 [ 598.030470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 598.038723] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 598.345030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 598.364759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 598.389619] LNet: Removed LNI 192.168.204.52@tcp232 [ 598.398733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 598.409592] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 598.683117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 598.702198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 598.718562] LNet: Removed LNI 192.168.204.52@tcp232 [ 598.724824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 598.733842] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 599.032791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 599.050930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 599.064617] LNet: Removed LNI 192.168.204.52@tcp232 [ 599.070403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 601.197817] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:06:04 (1755039964) [ 601.233630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.243894] LNet: 74719:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.245884] LNet: 74719:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 601.258174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.286641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 601.296233] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 601.298447] LNet: Accept secure, port 988 [ 601.551384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 601.577465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 601.602301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 601.627503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 601.642651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 601.656072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 603.806706] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:06:06 (1755039966) [ 603.847829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.867257] LNet: Removed LNI 192.168.204.52@tcp [ 603.882417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.903526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 603.912204] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 603.914341] LNet: Accept secure, port 988 [ 606.460485] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:06:09 (1755039969) [ 607.012518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 607.657098] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:06:10 (1755039970) [ 607.828165] LNet: Removed LNI 192.168.204.52@tcp [ 608.063476] Key type .llcrypt unregistered [ 608.064584] Key type ._llcrypt unregistered [ 610.090332] Key type ._llcrypt registered [ 610.091352] Key type .llcrypt registered [ 610.156354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 610.162644] alg: No test for adler32 (adler32-zlib) [ 610.985147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 610.997927] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 610.999565] LNet: Accept secure, port 988 [ 613.853142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 616.351369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.152@tcp [ 616.458180] LNet: 76888:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.465467] LNet: Removed LNI 192.168.204.52@tcp [ 616.617464] Key type .llcrypt unregistered [ 616.618437] Key type ._llcrypt unregistered [ 620.298937] Key type ._llcrypt registered [ 620.299901] Key type .llcrypt registered [ 620.784728] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 621.330533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 621.889713] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:06:24 (1755039984) [ 622.019936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 622.025211] alg: No test for adler32 (adler32-zlib) [ 622.830319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.860592] LNet: 77970:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.109428] Key type .llcrypt unregistered [ 623.110434] Key type ._llcrypt unregistered [ 625.145662] Key type ._llcrypt registered [ 625.147451] Key type .llcrypt registered [ 625.205892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 625.210766] alg: No test for adler32 (adler32-zlib) [ 626.013660] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 626.015105] LNet: 78221:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 626.017623] LNet: Accept secure, port 988 [ 626.021991] LNetError: 78233:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 626.025827] LNetError: 78233:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) route to o2ib through 192.168.204.56@tcp has gone from up to down [ 626.089599] LNetError: 78252:0:(router.c:334:lnet_is_route_alive()) peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 626.197228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.280114] LNetError: 78233:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) route to o2ib through 192.168.204.57@tcp has gone from up to down [ 629.287139] LNet: 78277:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.310410] LNet: Removed LNI 192.168.204.52@tcp [ 630.321256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.329741] LNet: 78280:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 632.286704] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:06:35 (1755039995) [ 632.958083] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg452-server [ 633.565303] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:06:36 (1755039996) [ 634.207411] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 634.296180] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 634.297459] LNet: 79288:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 634.299942] LNet: 79288:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 634.302184] LNet: Accept secure, port 988 [ 634.309123] LNetError: 79303:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 634.312649] LNetError: 79303:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) route to o2ib through 192.168.204.56@tcp has gone from up to down [ 634.315693] LNetError: 79303:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 635.888136] Key type lgssc registered [ 636.203168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 637.408404] LNetError: 79293:0:(lib-lnet.h:1312:lnet_set_route_aliveness()) route to o2ib through 192.168.204.54@tcp has gone from up to down [ 643.735828] Key type lgssc unregistered [ 643.855408] LNet: 80325:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.902643] LNet: Removed LNI 192.168.204.52@tcp [ 645.200648] Key type .llcrypt unregistered [ 645.201780] Key type ._llcrypt unregistered [ 650.357040] Key type ._llcrypt registered [ 650.358946] Key type .llcrypt registered [ 650.828974] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:06:53 (1755040013) [ 650.956778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 650.961588] alg: No test for adler32 (adler32-zlib) [ 651.833757] LNetError: 81164:0:(api-ni.c:2619:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 651.995313] Key type .llcrypt unregistered [ 651.996855] Key type ._llcrypt unregistered [ 653.727510] Key type ._llcrypt registered [ 653.728540] Key type .llcrypt registered [ 654.192652] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:06:57 (1755040017) [ 654.405237] Key type .llcrypt unregistered [ 654.406227] Key type ._llcrypt unregistered [ 656.443123] Key type ._llcrypt registered [ 656.444177] Key type .llcrypt registered [ 656.514710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 656.521059] alg: No test for adler32 (adler32-zlib) [ 659.375528] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:07:02 (1755040022) [ 659.404458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.417509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.444172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 659.455263] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 659.456826] LNet: Accept secure, port 988 [ 659.692951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 659.704072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 659.721325] LNet: Removed LNI 192.168.204.52@tcp [ 659.805620] LNet: 82741:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.966521] Key type .llcrypt unregistered [ 659.967486] Key type ._llcrypt unregistered [ 663.647361] Key type ._llcrypt registered [ 663.648607] Key type .llcrypt registered [ 664.118132] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:07:07 (1755040027) [ 664.229346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 664.234869] alg: No test for adler32 (adler32-zlib) [ 665.045978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.070650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 665.080054] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 665.082552] LNet: Accept secure, port 988 [ 665.327356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 665.338685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.52@tcp [ 665.344706] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 665.346242] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 665.347593] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 665, 0, 0 [ 665.349338] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 665.350399] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 665.351856] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 665.353178] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 665, 0, 0 [ 665.354876] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 665.355974] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 665.357490] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 665.358714] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 665, 0, 715 [ 665.360701] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 665.361720] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 665.363119] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 665.364464] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 665, 0, 715 [ 665.366263] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 667.598387] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:07:10 (1755040030) [ 667.780927] LNet: 84281:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.838679] LNet: Removed LNI 192.168.204.52@tcp [ 669.068238] Key type .llcrypt unregistered [ 669.069195] Key type ._llcrypt unregistered [ 671.080086] Key type ._llcrypt registered [ 671.081209] Key type .llcrypt registered [ 671.156138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 671.163161] alg: No test for adler32 (adler32-zlib) [ 671.974369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 671.984751] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 671.986338] LNet: Accept secure, port 988 [ 675.165202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 678.102216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.116869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 678.126647] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 678.487322] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 678.489551] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 678.519971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 678.527370] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 682.107682] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:07:25 (1755040045) [ 682.142101] LNet: 85733:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.168184] LNet: Removed LNI 192.168.204.52@tcp [ 684.192140] LNet: Removed LNI 192.168.204.52@tcp1 [ 684.200841] LNet: Removed LNI 10.1.2.3@tcp99 [ 684.443463] Key type .llcrypt unregistered [ 684.444600] Key type ._llcrypt unregistered [ 686.407636] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 686.409306] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 686.491640] Key type ._llcrypt registered [ 686.492770] Key type .llcrypt registered [ 686.553066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 686.558095] alg: No test for adler32 (adler32-zlib) [ 687.367634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.391054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.399550] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 687.401561] LNet: Accept secure, port 988 [ 687.650669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 687.658147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 688.342536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 688.358258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 688.373435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 688.381100] LNetError: 86450:0:(peer.c:2185:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 688.392899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 688.407097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 688.528325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 688.551516] LNet: 86470:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.568152] LNet: Removed LNI 192.168.204.52@tcp [ 689.578343] LNet: Removed LNI 10.1.2.3@tcp [ 689.808425] Key type .llcrypt unregistered [ 689.810371] Key type ._llcrypt unregistered [ 693.406976] Key type ._llcrypt registered [ 693.407985] Key type .llcrypt registered [ 693.892627] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:07:36 (1755040056) [ 694.018812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 694.023654] alg: No test for adler32 (adler32-zlib) [ 694.833434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.856723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.867390] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 694.868993] LNet: Accept secure, port 988 [ 695.126326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 695.158806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg452-client@tcp [ 697.320329] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:07:40 (1755040060) [ 697.357200] LNet: 87989:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.406948] LNet: Removed LNI 192.168.204.52@tcp [ 698.682335] Key type .llcrypt unregistered [ 698.683608] Key type ._llcrypt unregistered [ 700.561134] Key type ._llcrypt registered [ 700.562208] Key type .llcrypt registered [ 700.606444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 700.610959] alg: No test for adler32 (adler32-zlib) [ 701.403521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 701.413470] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 701.416079] LNet: Accept secure, port 988 [ 701.427496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 703.365885] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:07:46 (1755040066) [ 703.826884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 704.334220] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:07:47 (1755040067) [ 704.806982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 705.332441] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:07:48 (1755040068) [ 705.371443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.384544] LNet: 89405:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.394141] LNet: Removed LNI 192.168.204.52@tcp [ 705.403276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.423393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 705.433491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 705.642185] Key type .llcrypt unregistered [ 705.643298] Key type ._llcrypt unregistered [ 709.253950] Key type ._llcrypt registered [ 709.254919] Key type .llcrypt registered [ 709.760550] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:07:52 (1755040072) [ 709.889022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 709.896527] alg: No test for adler32 (adler32-zlib) [ 710.702459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.721697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 710.732494] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 710.735046] LNet: Accept secure, port 988 [ 710.987199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 711.007950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 713.243288] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:07:56 (1755040076) [ 713.275126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.284931] LNet: 90951:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.342730] LNet: Removed LNI 192.168.204.52@tcp [ 714.350918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.369020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 714.379849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 716.337259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 716.896832] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:07:59 (1755040079) [ 716.923805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.933376] LNet: 91668:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.943704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.022205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.033113] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 717.034538] LNet: Accept secure, port 988 [ 717.246038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 717.253458] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 717.556657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 717.574213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 717.600326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.52@tcp -m PUT -e local_timeout -r 1 [ 717.620989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 717.639790] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 717.641369] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 717.721857] LNetError: 91699:0:(lib-move.c:4909:lnet_parse()) 192.168.204.52@tcp, src 192.168.204.52@tcp: Dropping PUT (error -108 looking up sender) [ 718.752361] LNet: 91973:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.776120] LNet: Removed LNI 192.168.204.52@tcp [ 719.783449] LNet: Removed LNI 10.1.2.3@tcp [ 719.936647] Key type .llcrypt unregistered [ 719.938317] Key type ._llcrypt unregistered [ 723.701356] Key type ._llcrypt registered [ 723.702579] Key type .llcrypt registered [ 724.147299] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:08:07 (1755040087) [ 724.259449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 724.264746] alg: No test for adler32 (adler32-zlib) [ 725.068130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.183572] LNet: 92822:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.406287] Key type .llcrypt unregistered [ 725.407199] Key type ._llcrypt unregistered [ 727.354071] Key type ._llcrypt registered [ 727.355107] Key type .llcrypt registered [ 727.426869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.433575] alg: No test for adler32 (adler32-zlib) [ 728.234523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 728.244282] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 728.245938] LNet: Accept secure, port 988 [ 731.091750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 731.161389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 731.355055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.152@tcp [ 731.448203] LNet: 93246:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 731.470735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 731.482123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 731.660512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 731.770503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 731.787530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 731.979549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.152@tcp [ 732.063108] LNet: 93432:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 732.065405] LNet: 93432:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 732.088446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 732.098431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 732.279527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 732.320247] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 732.379090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 732.390416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 732.483396] LNet: 93566:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.542519] LNet: Removed LNI 192.168.204.52@tcp [ 733.717920] Key type .llcrypt unregistered [ 733.719525] Key type ._llcrypt unregistered [ 739.045112] Key type ._llcrypt registered [ 739.045982] Key type .llcrypt registered [ 739.496845] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:08:22 (1755040102) [ 739.623660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 739.628926] alg: No test for adler32 (adler32-zlib) [ 740.432526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.576781] LNet: 94449:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.829312] Key type .llcrypt unregistered [ 740.831063] Key type ._llcrypt unregistered [ 742.854937] Key type ._llcrypt registered [ 742.855946] Key type .llcrypt registered [ 742.919619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.926661] alg: No test for adler32 (adler32-zlib) [ 743.734983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 743.744214] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 743.746315] LNet: Accept secure, port 988 [ 746.574985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 748.994512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.004475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 749.010452] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 749.296068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 749.454754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.152@tcp [ 749.553327] LNet: 95071:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 749.583390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 749.597115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 749.814954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 749.920092] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.152@tcp, 192.168.204.152@tcp1 [ 749.926382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 749.937803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 750.153293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.152@tcp [ 750.254184] LNet: 95257:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 750.256870] LNet: 95257:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 750.281371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 750.294778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 750.510951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 750.626543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 750.638424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 752.660630] LNet: 95426:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.696125] LNet: Removed LNI 192.168.204.52@tcp [ 753.703293] LNet: Removed LNI 192.168.204.52@tcp1 [ 753.843493] Key type .llcrypt unregistered [ 753.844456] Key type ._llcrypt unregistered [ 758.671308] Key type ._llcrypt registered [ 758.672249] Key type .llcrypt registered [ 759.126755] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 644 sec =========== 19:08:42 (1755040122) [ 759.346395] Key type .llcrypt unregistered [ 759.347345] Key type ._llcrypt unregistered [ 763.204301] Key type ._llcrypt registered [ 763.206401] Key type .llcrypt registered [ 763.373185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 763.378647] alg: No test for adler32 (adler32-zlib) [ 764.223757] Lustre: Lustre: Build Version: 2.16.56_30_ga643f08 [ 764.302105] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 764.303653] LNet: Accept secure, port 988 [ 765.880166] Key type lgssc registered [ 766.264714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 787.549954] Lustre: Mounted lustre-client [ 789.788557] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 796.848699] LustreError: 98609:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a544320b800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 796.855209] LustreError: 98609:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 796.868153] Lustre: Unmounted lustre-client [ 823.054487] Key type lgssc unregistered [ 823.195639] LNet: 99088:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.231339] LNet: Removed LNI 192.168.204.52@tcp [ 824.617130] Key type .llcrypt unregistered [ 824.618419] Key type ._llcrypt unregistered