[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482370579 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] BRK [0x5a601000, 0x5a601fff] PGTABLE [ 0.000000] BRK [0x5a602000, 0x5a602fff] PGTABLE [ 0.000000] BRK [0x5a603000, 0x5a603fff] PGTABLE [ 0.000000] BRK [0x5a604000, 0x5a604fff] PGTABLE [ 0.000000] BRK [0x5a605000, 0x5a605fff] PGTABLE [ 0.000000] BRK [0x5a606000, 0x5a606fff] PGTABLE [ 0.000000] BRK [0x5a607000, 0x5a607fff] PGTABLE [ 0.000000] BRK [0x5a608000, 0x5a608fff] PGTABLE [ 0.000000] BRK [0x5a609000, 0x5a609fff] PGTABLE [ 0.000000] BRK [0x5a60a000, 0x5a60afff] PGTABLE [ 0.000000] BRK [0x5a60b000, 0x5a60bfff] PGTABLE [ 0.000000] BRK [0x5a60c000, 0x5a60cfff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] On node 0 totalpages: 1076600 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 158 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782298 pages, LIFO batch:63 [ 0.000000] Normal zone: 4536 pages used for memmap [ 0.000000] Normal zone: 290304 pages, LIFO batch:63 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] pcpu-alloc: s221184 r8192 d28672 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001000] hpet clockevent registered [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003236] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007451] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010155] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.013003] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.022384] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026524] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] *** VALIDATE tmpfs *** [ 0.030493] *** VALIDATE proc *** [ 0.031262] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.040584] debug: unmapping init [mem 0xffffffff90259000-0xffffffff90260fff] [ 0.042189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043776] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044028] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.049994] ... event mask: 000000070000000f [ 0.050309] rcu: Hierarchical SRCU implementation. [ 0.052514] smp: Bringing up secondary CPUs ... [ 0.053564] x86: Booting SMP configuration: [ 0.054023] .... node #0, CPUs: #1 #2 #3 [ 0.058011] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148024] node 0 deferred pages initialised in 85ms [ 0.151217] devtmpfs: initialized [ 0.153296] x86/mm: Memory block size: 128MB [ 0.156821] gcov: version magic: 0x41383552 [ 0.159138] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164325] pinctrl core: initialized pinctrl subsystem [ 0.166783] ************************************************************* [ 0.169013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171012] ** ** [ 0.173015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175013] ** ** [ 0.178014] ** This means that this kernel is built to expose internal ** [ 0.179010] ** IOMMU data structures, which may compromise security on ** [ 0.181014] ** your system. ** [ 0.184012] ** ** [ 0.186012] ** If you see this message and you are not debugging the ** [ 0.188025] ** kernel, report this immediately to your vendor! ** [ 0.190016] ** ** [ 0.193016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195014] ************************************************************* [ 0.197698] NET: Registered protocol family 16 [ 0.199440] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207123] cpuidle: using governor menu [ 0.208436] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211663] PCI: Using configuration type 1 for base access [ 0.213136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.226061] cryptd: max_cpu_qlen set to 1000 [ 0.230246] ACPI: Added _OSI(Module Device) [ 0.231013] ACPI: Added _OSI(Processor Device) [ 0.233012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238731] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243444] ACPI: Interpreter enabled [ 0.245062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246012] ACPI: Using IOAPIC for interrupt routing [ 0.247080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250380] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261864] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273347] acpiphp: Slot [2] registered [ 0.275110] acpiphp: Slot [5] registered [ 0.276108] acpiphp: Slot [6] registered [ 0.277114] acpiphp: Slot [3] registered [ 0.279115] acpiphp: Slot [4] registered [ 0.280118] acpiphp: Slot [7] registered [ 0.282121] acpiphp: Slot [8] registered [ 0.283113] acpiphp: Slot [9] registered [ 0.285078] acpiphp: Slot [10] registered [ 0.286136] acpiphp: Slot [11] registered [ 0.288116] acpiphp: Slot [12] registered [ 0.290227] acpiphp: Slot [13] registered [ 0.291115] acpiphp: Slot [14] registered [ 0.293119] acpiphp: Slot [15] registered [ 0.294157] acpiphp: Slot [16] registered [ 0.296096] acpiphp: Slot [17] registered [ 0.297277] acpiphp: Slot [18] registered [ 0.300021] acpiphp: Slot [19] registered [ 0.301135] acpiphp: Slot [20] registered [ 0.302084] acpiphp: Slot [21] registered [ 0.303103] acpiphp: Slot [22] registered [ 0.305104] acpiphp: Slot [23] registered [ 0.306129] acpiphp: Slot [24] registered [ 0.308135] acpiphp: Slot [25] registered [ 0.309105] acpiphp: Slot [26] registered [ 0.311152] acpiphp: Slot [27] registered [ 0.312111] acpiphp: Slot [28] registered [ 0.314114] acpiphp: Slot [29] registered [ 0.315117] acpiphp: Slot [30] registered [ 0.317117] acpiphp: Slot [31] registered [ 0.318127] PCI host bridge to bus 0000:00 [ 0.320025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.332039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333271] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335856] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339296] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.351056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.354023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361302] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.363701] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.366042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.368844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372917] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.383017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.387014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393055] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.458599] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.472016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.481019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.487670] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.489294] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.491327] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.493336] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.495174] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.500029] iommu: Default domain type: Passthrough [ 0.501333] SCSI subsystem initialized [ 0.502000] ACPI: bus type USB registered [ 0.503146] usbcore: registered new interface driver usbfs [ 0.505114] usbcore: registered new interface driver hub [ 0.507094] usbcore: registered new device driver usb [ 0.509176] pps_core: LinuxPPS API ver. 1 registered [ 0.510010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.513056] PTP clock support registered [ 0.515042] EDAC MC: Ver: 3.0.0 [ 0.516382] PCI: Using ACPI for IRQ routing [ 0.517010] PCI: pci_cache_line_size set to 64 bytes [ 0.517145] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.517155] e820: reserve RAM buffer [mem 0xbffda000-0xbfffffff] [ 0.517159] e820: reserve RAM buffer [mem 0x146e00000-0x147ffffff] [ 0.517634] NetLabel: Initializing [ 0.519012] NetLabel: domain hash size = 128 [ 0.520009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.521061] NetLabel: unlabeled traffic allowed by default [ 0.523134] vgaarb: loaded [ 0.524275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.526014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.531983] clocksource: Switched to clocksource kvm-clock [ 0.642980] VFS: Disk quotas dquot_6.6.0 [ 0.644704] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.647394] *** VALIDATE ramfs *** [ 0.648755] *** VALIDATE hugetlbfs *** [ 0.650418] pnp: PnP ACPI init [ 0.651864] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.651938] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.651967] pnp 00:02: [dma 2] [ 0.652069] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.652139] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.652219] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.652286] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.652894] pnp: PnP ACPI: found 6 devices [ 0.673235] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.676930] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.679391] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.681746] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.684283] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.686113] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.688621] NET: Registered protocol family 2 [ 0.691502] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.696724] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.700406] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.705482] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.708153] TCP: Hash tables configured (established 65536 bind 65536) [ 0.710664] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.713576] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.716336] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.719150] NET: Registered protocol family 1 [ 0.721571] RPC: Registered named UNIX socket transport module. [ 0.723597] RPC: Registered udp transport module. [ 0.725316] RPC: Registered tcp transport module. [ 0.727080] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.729397] NET: Registered protocol family 44 [ 0.731088] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.733239] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.735323] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.737780] PCI: CLS 0 bytes, default 64 [ 0.739369] Unpacking initramfs... [ 2.215708] debug: unmapping init [mem 0xffff95cf7cc64000-0xffff95cf7ffcffff] [ 2.222532] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.224764] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.228973] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.725659] Initialise system trusted keyrings [ 2.727178] Key type blacklist registered [ 2.729579] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.738066] zbud: loaded [ 2.741369] *** VALIDATE nfs *** [ 2.742468] *** VALIDATE nfs4 *** [ 2.743971] pstore: using deflate compression [ 2.747175] Platform Keyring initialized [ 2.881708] NET: Registered protocol family 38 [ 2.882965] Key type asymmetric registered [ 2.884137] Asymmetric key parser 'x509' registered [ 2.885483] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.887905] io scheduler mq-deadline registered [ 2.889246] io scheduler kyber registered [ 2.890413] io scheduler bfq registered [ 2.891904] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.893889] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.896253] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.898319] ACPI: Power Button [PWRF] [ 2.902624] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.909186] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.922191] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.950360] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.979394] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.985276] Non-volatile memory driver v1.3 [ 2.987307] Linux agpgart interface v0.103 [ 3.021573] virtio_blk virtio1: [vda] 146712 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.024614] vda: detected capacity change from 0 to 75116544 [ 3.039721] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.042425] vdb: detected capacity change from 0 to 1073741824 [ 3.048485] libphy: Fixed MDIO Bus: probed [ 3.063447] usbcore: registered new interface driver usbserial_generic [ 3.065682] usbserial: USB Serial support registered for generic [ 3.067553] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.071515] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.072968] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.075385] mousedev: PS/2 mouse device common for all mice [ 3.077395] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.077760] rtc_cmos 00:05: RTC can wake from S4 [ 3.082436] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.082693] rtc_cmos 00:05: registered as rtc0 [ 3.087905] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.088649] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.093987] intel_pstate: CPU model not supported [ 3.097675] hid: raw HID events driver (C) Jiri Kosina [ 3.100250] usbcore: registered new interface driver usbhid [ 3.102687] usbhid: USB HID core driver [ 3.105474] drop_monitor: Initializing network drop monitor service [ 3.108424] Initializing XFRM netlink socket [ 3.112896] NET: Registered protocol family 10 [ 3.116093] Segment Routing with IPv6 [ 3.117639] NET: Registered protocol family 17 [ 3.120643] mpls_gso: MPLS GSO support [ 3.122016] start plist test [ 3.124575] end plist test [ 3.126306] RAS: Correctable Errors collector initialized. [ 3.128484] AVX version of gcm_enc/dec engaged. [ 3.130459] AES CTR mode by8 optimization enabled [ 3.219900] sched_clock: Marking stable (3219866510, 0)->(4111378978, -891512468) [ 3.223369] registered taskstats version 1 [ 3.226273] Loading compiled-in X.509 certificates [ 3.228227] zswap: loaded using pool lzo/zbud [ 3.256589] Key type big_key registered [ 3.270304] Key type encrypted registered [ 3.272056] ima: No TPM chip found, activating TPM-bypass! [ 3.274222] ima: Allocated hash algorithm: sha1 [ 3.275840] ima: No architecture policies found [ 3.277131] evm: Initialising EVM extended attributes: [ 3.278930] evm: security.selinux [ 3.280081] evm: security.ima [ 3.281257] evm: security.capability [ 3.282406] evm: HMAC attrs: 0x1 [ 3.284242] rtc_cmos 00:05: setting system clock to 2026-09-08 04:57:19 UTC (1788843439) [ 3.290377] debug: unmapping init [mem 0xffffffff91203000-0xffffffff913fffff] [ 3.293175] debug: unmapping init [mem 0xffffffff8ff82000-0xffffffff90258fff] [ 3.300102] Write protecting the kernel read-only data: 28672k [ 3.303477] debug: unmapping init [mem 0xffffffff8e603000-0xffffffff8e7fffff] [ 3.305990] debug: unmapping init [mem 0xffffffff8ef14000-0xffffffff8effffff] [ 3.364249] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.371835] systemd[1]: Detected virtualization kvm. [ 3.373864] systemd[1]: Detected architecture x86-64. [ 3.375760] systemd[1]: Running in initial RAM disk. [ 3.400448] systemd[1]: No hostname configured. [ 3.402141] systemd[1]: Set hostname to . [ 3.403970] random: systemd: uninitialized urandom read (16 bytes read) [ 3.406367] systemd[1]: Initializing machine ID from random generator. [ 3.457162] random: ln: uninitialized urandom read (6 bytes read) [ 3.539536] random: systemd: uninitialized urandom read (16 bytes read) [ 3.542596] systemd[1]: Listening on udev Kernel Socket. [ 3.548288] systemd[1]: Reached target Local File Systems. [ 3.553512] systemd[1]: Reached target Initrd Root Device. [ 4.133481] device-mapper: uevent: version 1.0.3 [ 4.136892] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 4.727700] random: fast init done [ 4.770547] virtio_net virtio0 ens2: renamed from eth0 [ 4.854588] libata version 3.00 loaded. [ 4.900777] ata_piix 0000:00:01.1: version 2.13 [ 4.930917] scsi host0: ata_piix [ 4.950336] scsi host1: ata_piix [ 4.952478] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.954274] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.787925] random: crng init done [ 9.789441] random: 7 urandom warning(s) missed due to ratelimiting [ 10.120536] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 11.168781] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.405836] SELinux: Disabled at runtime. [ 11.464699] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.473559] systemd[1]: Detected virtualization kvm. [ 11.474930] systemd[1]: Detected architecture x86-64. [ 11.952741] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.955131] systemd[1]: Stopped Switch Root. [ 11.958984] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.961879] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.966218] systemd[1]: Stopped Journal Service. [ 11.972879] systemd[1]: Starting Journal Service... [ 11.976578] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ 12.062274] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 12.358127] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 12.649072] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.677324] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.752221] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.766843] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 12.766864] EDAC sbridge: Ver: 1.1.2 [ 13.723459] Key type dns_resolver registered [ 14.026470] NFS: Registering the id_resolver key type [ 14.028717] Key type id_resolver registered [ 14.030438] Key type id_legacy registered [ 43.511690] libcfs: loading out-of-tree module taints kernel. [ 43.574146] Key type ._llcrypt registered [ 43.575894] Key type .llcrypt registered [ 43.862280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 43.869261] alg: No test for adler32 (adler32-zlib) [ 44.876234] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 45.275702] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 46.936252] Key type lgssc registered [ 49.691515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.633023] hrtimer: interrupt took 2060830 ns [ 204.081686] Lustre: Mounted lustre-client - version 2.17.57_111_g33f3974 [ 209.771649] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.715525] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing check_logdir /tmp/testlogs/ [ 229.861265] Lustre: lustre-OST0000-osc-ffff95cfc99df800: disconnect after 23s idle [ 230.316749] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing yml_node [ 236.351697] Lustre: DEBUG MARKER: Client: 2.17.57.111 [ 239.275842] Lustre: DEBUG MARKER: MDS: 2.17.57.111 [ 241.965033] Lustre: DEBUG MARKER: OSS: 2.17.57.111 [ 244.161355] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 8 01:01:18 EDT 2026 [ 261.834576] Lustre: Unmounted lustre-client [ 289.861848] Key type lgssc unregistered [ 290.170402] LNet: 8431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.178791] LNetError: 8431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.205024] LNet: Removed LNI 192.168.202.29@tcp [ 291.121271] Key type .llcrypt unregistered [ 291.124572] Key type ._llcrypt unregistered [ 322.036217] Key type ._llcrypt registered [ 322.039284] Key type .llcrypt registered [ 322.413603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 322.446579] alg: No test for adler32 (adler32-zlib) [ 323.375621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 323.467750] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 323.492383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 328.535589] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 328.624249] LNet: 9302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.640921] LNetError: 9302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 328.660815] LNet: Removed LNI 192.168.202.29@tcp [ 329.379392] Key type .llcrypt unregistered [ 329.381236] Key type ._llcrypt unregistered [ 336.542919] Key type ._llcrypt registered [ 336.545643] Key type .llcrypt registered [ 338.290155] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:02:52 (1788843772) [ 338.680558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 338.697230] alg: No test for adler32 (adler32-zlib) [ 339.668398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.754329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.773927] LNet: 9720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.794601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.882655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.932598] LNet: 9727:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 347.987196] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:03:02 (1788843782) [ 348.110730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.169331] LNet: 10300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.228253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.400888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 355.343650] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:03:09 (1788843789) [ 355.452520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.497244] LNet: 10886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.554702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.687872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 363.204937] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:03:17 (1788843797) [ 363.508291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.536862] LNet: 11478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.551553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.608349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 370.168894] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:03:24 (1788843804) [ 370.280434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.321749] LNet: 12057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.358092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.426605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 370.703326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 377.579876] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:03:32 (1788843812) [ 377.845695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.908533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.990522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 378.373765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 378.784360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 385.733191] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:03:40 (1788843820) [ 386.061579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.099598] LNet: 13252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.120263] LNet: 13252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 386.143599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.222464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 393.225409] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:03:48 (1788843828) [ 393.421548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.495169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.552449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 393.586884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.113167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 394.144699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 394.600123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 394.639824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 395.102933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 395.141469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 395.583835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 395.615652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.159444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 396.229329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.591398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 396.631332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.037877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 397.060612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 397.452426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 397.505410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 397.939328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 397.992624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 398.541331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 398.582181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 399.081904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 399.119113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 399.634800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 399.671392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 407.045438] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:04:01 (1788843841) [ 407.315333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.345980] LNet: 14519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.350469] LNet: 14519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 407.368576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.428825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 407.459449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 415.652051] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:04:10 (1788843850) [ 416.074854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.139243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.217363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 416.282666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 422.358216] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:04:17 (1788843857) [ 422.703326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.740887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.813944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 422.852866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 429.882804] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:04:24 (1788843864) [ 430.218585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.261303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.339894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 430.390410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 437.147580] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:04:32 (1788843872) [ 437.351893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.377935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.428980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 437.463501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 444.827348] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:04:39 (1788843879) [ 445.009614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.055120] LNet: 17448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.068076] LNet: 17448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 445.115357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.213574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 445.283960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 453.018961] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:04:47 (1788843887) [ 453.177021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.242565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.355595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.936446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.758619] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:04:55 (1788843895) [ 461.039329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.127661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.358145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.614782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.045817] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:05:03 (1788843903) [ 469.249553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.317673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.461621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.638699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.234131] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:05:12 (1788843912) [ 478.355141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.447204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.084717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.427908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 486.501898] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:05:21 (1788843921) [ 486.822881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.880759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.965162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 487.025323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.928483] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:05:28 (1788843928) [ 494.042865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.079366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.152840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 494.192682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.041913] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:05:36 (1788843936) [ 502.350863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.414830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.531709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 512.238737] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:05:46 (1788843946) [ 512.336715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.376660] LNet: 22300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.386896] LNet: 22300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 512.432965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.533385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 522.179587] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:05:56 (1788843956) [ 522.466391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.536849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.633446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 523.127471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 531.029689] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:06:05 (1788843965) [ 531.183910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.252082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.416804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 531.472196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 539.908623] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:06:14 (1788843974) [ 540.020343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.128350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.271784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 540.315429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 548.202267] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:06:22 (1788843982) [ 548.302846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.377679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.517950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 548.574625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 556.297876] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:06:30 (1788843990) [ 556.458815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.538096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.644795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 556.680132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 564.177672] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:06:38 (1788843998) [ 564.292549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.366463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.449300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 564.579353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 564.609900] LNetError: 25844:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 564.698289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 564.764583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 572.873676] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:06:47 (1788844007) [ 573.050278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.166981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.316157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QsARuPkY2w [ 573.452842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QsARuPkY2w [ 573.525820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QsARuPkY2w [ 582.693610] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:06:56 (1788844016) [ 583.291776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.361697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.475588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 583.530038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 583.605600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 591.837853] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:07:06 (1788844026) [ 592.005840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.083689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.148969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.176505] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 593.362687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 594.953672] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 596.960185] Key type lgssc registered [ 598.270893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.573570] LNetError: 27606:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.29@tcp99 has no available nets [ 604.052664] Key type lgssc unregistered [ 604.404840] LNetError: 28712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.480750] LNet: Removed LNI 192.168.202.29@tcp [ 606.317986] Key type .llcrypt unregistered [ 606.329119] Key type ._llcrypt unregistered [ 612.561287] Key type ._llcrypt registered [ 612.565718] Key type .llcrypt registered [ 613.994432] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:07:28 (1788844048) [ 614.545493] Key type .llcrypt unregistered [ 614.548390] Key type ._llcrypt unregistered [ 620.869751] Key type ._llcrypt registered [ 620.873333] Key type .llcrypt registered [ 620.981829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 620.996030] alg: No test for adler32 (adler32-zlib) [ 621.892102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.964312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 621.979919] LNet: 29622:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 621.984879] LNet: 29622:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 621.990273] LNet: 29622:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 622.002171] LNet: Message forwarding will be enabled in 2s [ 625.056249] LNet: Message forwarding enabled [ 625.551844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 625.577879] LNet: Message forwarding will stop in 2s to 6s [ 629.155526] LNet: Message forwarding disabled [ 630.238947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 630.268105] LNet: 29721:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 630.284460] LNet: 29721:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 630.292968] LNet: 29721:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 630.307413] LNet: Message forwarding will be enabled in 2s [ 630.480265] LNet: 29734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.193037] Key type .llcrypt unregistered [ 631.194781] Key type ._llcrypt unregistered [ 643.417609] Key type ._llcrypt registered [ 643.419414] Key type .llcrypt registered [ 644.815933] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:07:59 (1788844079) [ 645.484521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 645.506719] alg: No test for adler32 (adler32-zlib) [ 646.433454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.513855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 654.209327] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:08:08 (1788844088) [ 654.430639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.479663] LNet: 31127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.531753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.668586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 654.705870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 654.733391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 654.795439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 654.846732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 654.875546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 654.909846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 654.942099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 654.983635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 655.024935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 655.062321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 655.106247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 655.146950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 655.181479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 655.211608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 655.242768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 655.274873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 655.306840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 655.338810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 655.357378] LNetError: 31178:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 661.854885] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:08:16 (1788844096) [ 662.145757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.172177] LNet: 31749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.192262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.302816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.014531] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:08:24 (1788844104) [ 672.076892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg229-server [ 674.190578] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:08:29 (1788844109) [ 674.369522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.418467] LNet: 32682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.456742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.535668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.558571] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 675.387893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[30-34]@tcp [ 675.406683] LNet: 32815:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 675.466174] LNetError: 32686:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.31@tcp is being used as a gateway but routing feature is not turned on [ 675.491160] LNetError: 32686:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.31@tcp has gone from up to down [ 681.945672] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:08:37 (1788844117) [ 683.759221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg229-server [ 686.228085] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:08:40 (1788844120) [ 686.355489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.401355] LNet: 33744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.409310] LNetError: 33744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.470129] LNet: Removed LNI 192.168.202.29@tcp [ 687.541538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.612821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.671911] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 688.417540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[30-35/2]@tcp [ 688.441739] LNet: 33870:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 688.463617] LNet: 33870:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 688.497437] LNetError: 33757:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.30@tcp is being used as a gateway but routing feature is not turned on [ 688.512303] LNetError: 33757:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 688.524362] LNetError: 33757:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.30@tcp has gone from up to down [ 688.532240] LNetError: 33757:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 688.543176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[30-35/2]@tcp [ 695.484620] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:08:50 (1788844130) [ 695.810258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.851227] LNet: 34463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.866517] LNetError: 34463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 695.895217] LNet: Removed LNI 192.168.202.29@tcp [ 695.937229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.089393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 696.179673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.233498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.322472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.400795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.443456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.491874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 696.580499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.621529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.715236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.811274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.855658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.922485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 697.023259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.064689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.169966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.287964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.324882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.379875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 697.481707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.537605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.645065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.774309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.819591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.872975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 697.940423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.977679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.042568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.111871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.142689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.185380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 698.248821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.295152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.363436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 705.415317] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:09:00 (1788844140) [ 707.268383] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg229-server [ 709.349265] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:09:04 (1788844144) [ 711.787702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg229-server [ 713.582548] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:09:08 (1788844148) [ 713.806347] LNet: 35861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.814194] LNet: 35861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 714.489859] Key type .llcrypt unregistered [ 714.500332] Key type ._llcrypt unregistered [ 721.571591] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 721.787261] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 721.790516] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 721.796752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 724.373379] Key type ._llcrypt registered [ 724.374927] Key type .llcrypt registered [ 724.594679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.618921] alg: No test for adler32 (adler32-zlib) [ 725.639344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.717534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.741057] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 726.690563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 726.714596] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.815851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 729.837723] LNet: Removed LNI 10.1.2.3@tcp [ 729.989595] LNet: 36575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.994992] LNetError: 36575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.009799] LNet: Removed LNI 192.168.202.29@tcp [ 730.851899] Key type .llcrypt unregistered [ 730.854786] Key type ._llcrypt unregistered [ 743.348542] Key type ._llcrypt registered [ 743.350791] Key type .llcrypt registered [ 744.646550] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:09:39 (1788844179) [ 745.376541] Key type .llcrypt unregistered [ 745.380241] Key type ._llcrypt unregistered [ 751.190156] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 751.414829] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 751.418537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 751.421494] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 754.115885] Key type ._llcrypt registered [ 754.121708] Key type .llcrypt registered [ 754.379312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.401786] alg: No test for adler32 (adler32-zlib) [ 755.339359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.379707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.402521] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 756.253415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.283077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 758.661555] LNet: 38064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.667176] LNetError: 38064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 758.673739] LNet: Removed LNI 10.1.2.3@tcp [ 758.687961] LNet: Removed LNI 192.168.202.29@tcp [ 759.523680] Key type .llcrypt unregistered [ 759.529740] Key type ._llcrypt unregistered [ 770.772681] Key type ._llcrypt registered [ 770.775237] Key type .llcrypt registered [ 772.240317] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:10:06 (1788844206) [ 772.737645] Key type .llcrypt unregistered [ 772.746576] Key type ._llcrypt unregistered [ 778.750427] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.045750] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.053713] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.058600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 781.744385] Key type ._llcrypt registered [ 781.746563] Key type .llcrypt registered [ 781.932295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 781.964760] alg: No test for adler32 (adler32-zlib) [ 782.979392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.057681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 783.093984] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 785.690682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 785.729849] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 788.397924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 788.442433] LNet: Removed LNI 10.1.2.3@tcp [ 788.466453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 788.502470] LNet: Removed LNI 10.1.2.31@tcp [ 788.535418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 788.559356] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 789.511871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 789.543442] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 790.553904] LNet: 39980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.560482] LNetError: 39980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 790.566104] LNet: Removed LNI 10.1.2.31@tcp [ 790.580468] LNet: Removed LNI 10.1.2.3@tcp [ 791.380929] Key type .llcrypt unregistered [ 791.385089] Key type ._llcrypt unregistered [ 803.095040] Key type ._llcrypt registered [ 803.100891] Key type .llcrypt registered [ 804.314468] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:10:39 (1788844239) [ 804.938128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.956276] alg: No test for adler32 (adler32-zlib) [ 805.866515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.949883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.984477] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 806.680382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.706672] LNet: 40930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.712624] LNetError: 40930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.736637] LNet: Removed LNI 192.168.202.29@tcp [ 806.753526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.822524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.29@tcp [ 806.859708] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 807.709244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.29@tcp [ 807.739397] LNet: Removed LNI 192.168.202.29@tcp [ 807.803632] LNet: 41065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.386957] Key type .llcrypt unregistered [ 808.390463] Key type ._llcrypt unregistered [ 820.448262] Key type ._llcrypt registered [ 820.450285] Key type .llcrypt registered [ 821.622730] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:10:56 (1788844256) [ 823.457522] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg229-server [ 825.301934] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:11:00 (1788844260) [ 825.991300] Key type .llcrypt unregistered [ 825.993504] Key type ._llcrypt unregistered [ 832.494455] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 832.775254] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 832.789339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 832.793317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 835.401261] Key type ._llcrypt registered [ 835.403257] Key type .llcrypt registered [ 835.584395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.595491] alg: No test for adler32 (adler32-zlib) [ 836.500241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.575247] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 837.251206] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 839.073990] LNet: 42897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.081341] LNetError: 42897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.085389] LNet: Removed LNI 192.168.202.29@tcp1 [ 839.094237] LNet: Removed LNI 10.1.2.3@tcp2 [ 839.742518] Key type .llcrypt unregistered [ 839.745279] Key type ._llcrypt unregistered [ 852.321668] Key type ._llcrypt registered [ 852.326249] Key type .llcrypt registered [ 853.635299] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:11:28 (1788844288) [ 854.234921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 854.256829] alg: No test for adler32 (adler32-zlib) [ 855.239587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.331653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 855.379831] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 856.726496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.758373] LNet: 43864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.768187] LNetError: 43864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.790703] LNet: Removed LNI 192.168.202.29@tcp [ 856.816348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.891113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 856.924357] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 865.473672] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:11:39 (1788844299) [ 865.545162] LNet: 44448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.554107] LNetError: 44448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.573315] LNet: Removed LNI 192.168.202.29@tcp [ 866.306492] Key type .llcrypt unregistered [ 866.308123] Key type ._llcrypt unregistered [ 871.393570] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 871.513990] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 871.529697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 871.533901] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 874.280757] Key type ._llcrypt registered [ 874.284092] Key type .llcrypt registered [ 874.514686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 874.525167] alg: No test for adler32 (adler32-zlib) [ 875.428705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.488474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.510175] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 876.420626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 876.438812] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 879.163730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.182306] LNet: 45197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.187979] LNetError: 45197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 879.200252] LNet: Removed LNI 10.1.2.3@tcp [ 879.213801] LNet: Removed LNI 192.168.202.29@tcp [ 879.241331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.329616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 879.363475] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 879.367436] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 879.939865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.977397] LNet: 45224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.988232] LNetError: 45224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 879.993989] LNetError: 45224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 880.013687] LNet: Removed LNI 10.1.2.3@tcp [ 880.032202] LNet: Removed LNI 192.168.202.29@tcp [ 880.064592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.243355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 880.307066] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 880.314599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 881.059645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 881.106289] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 883.110688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.141919] LNet: 45386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.151443] LNetError: 45386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.157450] LNetError: 45386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 883.165109] LNet: Removed LNI 10.1.2.3@tcp [ 883.169460] LNet: Removed LNI 192.168.202.29@tcp [ 883.183950] LNet: Removed LNI 192.168.202.29@tcp2 [ 883.237123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.357936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 883.394143] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 883.405862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.411031] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 894.334936] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:12:08 (1788844328) [ 894.735785] LNet: 45984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.753216] LNetError: 45984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 894.761313] LNetError: 45984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 894.767695] LNet: Removed LNI 10.1.2.3@tcp [ 894.769690] LNet: Removed LNI 192.168.202.29@tcp [ 894.788348] LNet: Removed LNI 192.168.202.29@tcp2 [ 895.613441] Key type .llcrypt unregistered [ 895.617225] Key type ._llcrypt unregistered [ 902.132758] Key type ._llcrypt registered [ 902.137705] Key type .llcrypt registered [ 902.489429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.515254] alg: No test for adler32 (adler32-zlib) [ 903.465705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.507723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.547841] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 904.313780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.345496] LNet: 46377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.354510] LNetError: 46377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.374796] LNet: Removed LNI 192.168.202.29@tcp [ 904.423421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.528366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 904.577255] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 905.159736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.190281] LNet: 46408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.202339] LNetError: 46408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.215854] LNet: Removed LNI 192.168.202.29@tcp [ 905.237984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.304745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 905.327118] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 905.965148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.013489] LNet: Removed LNI 192.168.202.29@tcp [ 906.037371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.119850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 906.138546] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 906.760823] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 906.907459] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 906.914700] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 906.922809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 909.473465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.492553] LNet: 46516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.499413] LNet: 46516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 909.508312] LNetError: 46516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.512685] LNetError: 46516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 909.535246] LNet: Removed LNI 192.168.202.29@tcp [ 909.577522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.651134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 909.693095] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 917.402820] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:12:31 (1788844351) [ 917.660605] LNet: 47114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.668915] LNetError: 47114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.685810] LNet: Removed LNI 192.168.202.29@tcp [ 918.275853] Key type .llcrypt unregistered [ 918.283967] Key type ._llcrypt unregistered [ 924.709280] Key type ._llcrypt registered [ 924.711626] Key type .llcrypt registered [ 924.908168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 924.921442] alg: No test for adler32 (adler32-zlib) [ 925.787823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.829937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.850745] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 926.568819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.584691] LNet: 47522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.587952] LNetError: 47522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.603934] LNet: Removed LNI 192.168.202.29@tcp [ 926.618373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.677136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 926.700739] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 932.994533] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:12:47 (1788844367) [ 933.165345] LNet: 48113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.176301] LNetError: 48113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.193943] LNet: Removed LNI 192.168.202.29@tcp [ 933.626623] Key type .llcrypt unregistered [ 933.630246] Key type ._llcrypt unregistered [ 939.167372] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 939.431832] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 939.446711] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 939.463770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 942.133983] Key type ._llcrypt registered [ 942.136474] Key type .llcrypt registered [ 942.391909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.408812] alg: No test for adler32 (adler32-zlib) [ 943.285186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.325779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.342681] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 944.238543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.252861] LNet: 48561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.258699] LNetError: 48561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.271052] LNet: Removed LNI 192.168.202.29@tcp [ 944.289189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.329917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 944.352975] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 951.522544] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:13:06 (1788844386) [ 951.616345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.656114] LNet: 49154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.669152] LNetError: 49154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.712108] LNet: Removed LNI 192.168.202.29@tcp [ 951.814452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.968449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 958.571462] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:13:12 (1788844392) [ 958.663891] LNet: 49732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.401833] Key type .llcrypt unregistered [ 959.405183] Key type ._llcrypt unregistered [ 965.933798] Key type ._llcrypt registered [ 965.937532] Key type .llcrypt registered [ 966.113344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.133576] alg: No test for adler32 (adler32-zlib) [ 967.122026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.179664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.207671] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 968.101241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.133787] LNet: 50135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.138480] LNetError: 50135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.154847] LNet: Removed LNI 192.168.202.29@tcp [ 968.173762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.243903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 968.273406] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 975.300940] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:13:29 (1788844409) [ 975.403516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.422560] LNet: 50722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.427026] LNetError: 50722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.437336] LNet: Removed LNI 192.168.202.29@tcp [ 975.458030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.551568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 981.773516] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:13:36 (1788844416) [ 983.136514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 984.446816] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:13:39 (1788844419) [ 985.765472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 987.414463] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:13:42 (1788844422) [ 989.081480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 991.097393] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:13:45 (1788844425) [ 992.775792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 994.377196] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:13:49 (1788844429) [ 995.941236] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 997.581776] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:13:52 (1788844432) [ 997.856147] LNet: 52782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.437719] Key type .llcrypt unregistered [ 998.440525] Key type ._llcrypt unregistered [ 1004.917799] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1005.160496] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1005.176818] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1005.202180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1007.792475] Key type ._llcrypt registered [ 1007.794323] Key type .llcrypt registered [ 1007.937795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1007.957377] alg: No test for adler32 (adler32-zlib) [ 1008.892810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.978755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1009.004912] LNet: Added LNI 192.168.202.29@tcp [8/512/0/180] [ 1009.009751] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1009.012500] LNet: 53100:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1009.106199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.126992] LNet: 53119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.135398] LNetError: 53119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1009.139949] LNet: Removed LNI 10.1.2.3@tcp [ 1009.142517] LNetError: 53119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.158487] LNet: Removed LNI 192.168.202.29@tcp [ 1009.186529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.270784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1009.291511] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1009.370035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.388878] LNet: 53143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.403579] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.422539] LNet: Removed LNI 192.168.202.29@tcp [ 1009.442075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.515853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1009.538249] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1009.542060] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.546182] LNet: 53151:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1009.649908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.693477] LNet: 53169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.702155] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1009.715319] LNet: Removed LNI 10.1.2.3@tcp [ 1009.716882] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.736717] LNet: Removed LNI 192.168.202.29@tcp [ 1009.767951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.837537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1009.854516] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1009.859803] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.862667] LNet: 53177:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1010.035888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.053731] LNet: 53203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.058852] LNetError: 53203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1010.063777] LNet: Removed LNI 10.1.2.3@tcp [ 1010.066739] LNetError: 53203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.077718] LNet: Removed LNI 192.168.202.29@tcp [ 1010.092154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.147585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1010.164040] LNet: Added LNI 192.168.202.29@tcp [8/256/0/182] [ 1010.168753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1010.253860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.269404] LNet: 53231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.275782] LNetError: 53231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1010.280181] LNet: Removed LNI 10.1.2.3@tcp [ 1010.283558] LNetError: 53231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.297886] LNet: Removed LNI 192.168.202.29@tcp [ 1010.314700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.372816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1010.396995] LNet: Added LNI 192.168.202.29@tcp [8/256/0/182] [ 1010.403081] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1010.406757] LNet: 53239:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1010.487765] LNet: Removed LNI 10.1.2.3@tcp [ 1010.515131] LNet: Removed LNI 192.168.202.29@tcp [ 1011.051953] Key type .llcrypt unregistered [ 1011.054340] Key type ._llcrypt unregistered [ 1023.042629] Key type ._llcrypt registered [ 1023.046414] Key type .llcrypt registered [ 1024.290756] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:14:19 (1788844459) [ 1026.056893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1027.788295] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:14:22 (1788844462) [ 1028.229527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.253877] alg: No test for adler32 (adler32-zlib) [ 1029.187186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.268564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.310903] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1030.282577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.29 [ 1030.295560] LNet: 54501:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1030.303554] LNet: Removed LNI 192.168.202.29@tcp [ 1036.256358] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:14:31 (1788844471) [ 1036.379454] LNet: 55065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.388150] LNetError: 55065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.402165] LNet: Removed LNI 192.168.202.29@tcp [ 1036.918473] Key type .llcrypt unregistered [ 1036.920622] Key type ._llcrypt unregistered [ 1043.193299] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1043.376522] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1046.082738] Key type ._llcrypt registered [ 1046.084385] Key type .llcrypt registered [ 1046.282547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1046.308980] alg: No test for adler32 (adler32-zlib) [ 1047.676857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1055.899663] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:14:50 (1788844490) [ 1056.129163] LNet: 55979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.137751] LNetError: 55979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1056.190715] LNet: Removed LNI 10.1.2.3@tcp [ 1057.068681] Key type .llcrypt unregistered [ 1057.074429] Key type ._llcrypt unregistered [ 1065.957940] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1066.284996] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1066.294428] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1066.306611] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1069.430327] Key type ._llcrypt registered [ 1069.436577] Key type .llcrypt registered [ 1069.763267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1069.789821] alg: No test for adler32 (adler32-zlib) [ 1070.787847] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1078.017306] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:15:13 (1788844513) [ 1079.974119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1081.797494] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:15:16 (1788844516) [ 1082.132155] LNet: 57162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.137439] LNetError: 57162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1082.152630] LNet: Removed LNI 10.1.2.3@tcp [ 1082.722602] Key type .llcrypt unregistered [ 1082.723984] Key type ._llcrypt unregistered [ 1089.426779] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1089.642665] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1089.647435] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1089.650109] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1092.328782] Key type ._llcrypt registered [ 1092.331510] Key type .llcrypt registered [ 1092.562027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.603317] alg: No test for adler32 (adler32-zlib) [ 1093.492736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.554395] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1101.321962] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:15:36 (1788844536) [ 1101.962869] LNet: 58388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.967283] LNetError: 58388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1101.981428] LNet: Removed LNI 10.1.2.3@tcp [ 1102.745593] Key type .llcrypt unregistered [ 1102.749890] Key type ._llcrypt unregistered [ 1108.583954] Key type ._llcrypt registered [ 1108.586661] Key type .llcrypt registered [ 1108.960646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.985934] alg: No test for adler32 (adler32-zlib) [ 1109.989676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1110.041814] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1118.257750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1118.820755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_interrupt -r 1 [ 1118.852347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_interrupt -r 1 [ 1118.880348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1118.895978] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1119.083119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.118853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.147328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.175714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.203715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.238985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.264534] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1119.271485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.302865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.327722] LNet: 58822:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1119.350478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.383119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.432799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.464455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.500865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.535853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.599567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.189700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_dropped -r 1 [ 1120.227619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_dropped -r 1 [ 1120.266544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1120.309152] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1120.322542] LNet: Skipped 1 previous similar message [ 1120.536681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.563941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.599809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.639674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.670602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.715433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.772889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.805267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.830284] LNet: 58931:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.845329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.866359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.897615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.922545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.950055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.969921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.016641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1121.611298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_aborted -r 1 [ 1121.654159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_aborted -r 1 [ 1121.688037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1121.720535] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1121.931164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.966319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.997848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.020660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.043430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.065740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.100819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.140668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.159511] LNet: 59041:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.176824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.206477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.232249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.254699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.277270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.310145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.343718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.887751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_no_route -r 1 [ 1122.911366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_no_route -r 1 [ 1122.938603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1123.249524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.289263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1123.338213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.373590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.409419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.449353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.477889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.510566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.544996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.572865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.601989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.631923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.659803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.692444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.719896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.283449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_timeout -r 1 [ 1124.320395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_timeout -r 1 [ 1124.354477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1124.371428] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1124.380222] LNet: Skipped 1 previous similar message [ 1124.609650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.643918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.686735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.720386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.754921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.805733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.841774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.885363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.903279] LNet: 59259:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.910137] LNet: 59259:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1124.934765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.999891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.042927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.106527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.152686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.216411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.260243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1125.735586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_error -r 1 [ 1125.784734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_error -r 1 [ 1125.809729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1126.012444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.037565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.061495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.088324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.117905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.146329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.169443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.197280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.218534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.243695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.266953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.291629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.316694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.355348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.382389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1126.600074] LNet: 59409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.605980] LNetError: 59409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.657463] LNet: Removed LNI 192.168.202.29@tcp [ 1128.168162] Key type .llcrypt unregistered [ 1128.170311] Key type ._llcrypt unregistered [ 1147.589323] Key type ._llcrypt registered [ 1147.596751] Key type .llcrypt registered [ 1148.971824] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:16:23 (1788844583) [ 1149.866647] Key type .llcrypt unregistered [ 1149.869631] Key type ._llcrypt unregistered [ 1158.314701] Key type ._llcrypt registered [ 1158.320123] Key type .llcrypt registered [ 1158.566318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.587291] alg: No test for adler32 (adler32-zlib) [ 1159.505659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1159.529778] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1166.865197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1178.889051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.944623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1178.991894] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1180.662326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_interrupt -r 1 [ 1180.702152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_interrupt -r 1 [ 1180.747763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_interrupt -r 1 [ 1180.802207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_interrupt -r 1 [ 1180.863334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_interrupt -r 1 [ 1180.913727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_interrupt -r 1 [ 1180.959840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_interrupt -r 1 [ 1181.022800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_interrupt -r 1 [ 1181.074921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1181.114092] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1181.129378] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1181.408798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.430521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.455289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.481866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.508442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.536507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.570281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.607299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.623170] LNet: 60874:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.669375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.713437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.776609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.820241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.856044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.905943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.971787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.657892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_dropped -r 1 [ 1182.710910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_dropped -r 1 [ 1182.741713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_dropped -r 1 [ 1182.780973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_dropped -r 1 [ 1182.830269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_dropped -r 1 [ 1182.880550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_dropped -r 1 [ 1182.922606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_dropped -r 1 [ 1182.968383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_dropped -r 1 [ 1183.016431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1183.036859] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1183.041259] LNet: Skipped 2 previous similar messages [ 1183.275977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.314248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.340453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.363423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.389296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.432949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.470314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.503917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.532799] LNet: 60995:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.557252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.590599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.642289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.670591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.715266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.754955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.802941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.328129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_aborted -r 1 [ 1184.357568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_aborted -r 1 [ 1184.384726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_aborted -r 1 [ 1184.416326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_aborted -r 1 [ 1184.443571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_aborted -r 1 [ 1184.469244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_aborted -r 1 [ 1184.495547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_aborted -r 1 [ 1184.526236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_aborted -r 1 [ 1184.562945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1184.620568] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1184.624030] LNet: Skipped 2 previous similar messages [ 1184.857930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.887054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.965703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.018765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.078577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.138541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.168440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.194454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.210487] LNet: 61116:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.225863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.253987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.280694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.307554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.336354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.361768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.385156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.992035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_no_route -r 1 [ 1186.034403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_no_route -r 1 [ 1186.080068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_no_route -r 1 [ 1186.125560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_no_route -r 1 [ 1186.171416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_no_route -r 1 [ 1186.259571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_no_route -r 1 [ 1186.309081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_no_route -r 1 [ 1186.384823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_no_route -r 1 [ 1186.423537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1186.710744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.733747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.756370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.785699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.818439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.859219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.896890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.922194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.953847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.985991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.032664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.093970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.138722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.178967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.223696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1187.815732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_timeout -r 1 [ 1187.843871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_timeout -r 1 [ 1187.874693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_timeout -r 1 [ 1187.907705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_timeout -r 1 [ 1187.939474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_timeout -r 1 [ 1187.990343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_timeout -r 1 [ 1188.033042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_timeout -r 1 [ 1188.069705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_timeout -r 1 [ 1188.107868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1188.134385] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1188.143251] LNet: Skipped 5 previous similar messages [ 1188.376918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.414021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.445754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.466929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.486343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.506689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.536271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.564310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.582852] LNet: 61359:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1188.586951] LNet: 61359:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1188.600734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.625166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.661677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.686827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.712978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.740826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1188.769746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1189.258967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_error -r 1 [ 1189.291326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_error -r 1 [ 1189.323705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_error -r 1 [ 1189.363677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_error -r 1 [ 1189.408902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_error -r 1 [ 1189.448256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_error -r 1 [ 1189.495773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_error -r 1 [ 1189.543854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_error -r 1 [ 1189.583695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1189.915989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.952703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1190.001900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.039790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.071850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.102336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.132446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.157471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.196938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.229532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.274422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.308277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.344840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.366752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.415050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1197.287413] LNet: 61556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.293287] LNetError: 61556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.314559] LNet: Removed LNI 192.168.202.29@tcp [ 1198.394815] LNet: Removed LNI 192.168.202.29@tcp1 [ 1198.985652] Key type .llcrypt unregistered [ 1198.989584] Key type ._llcrypt unregistered [ 1219.349253] Key type ._llcrypt registered [ 1219.369376] Key type .llcrypt registered [ 1221.424664] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:17:35 (1788844655) [ 1222.518281] Key type .llcrypt unregistered [ 1222.521482] Key type ._llcrypt unregistered [ 1230.666763] Key type ._llcrypt registered [ 1230.668293] Key type .llcrypt registered [ 1230.933692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1230.961503] alg: No test for adler32 (adler32-zlib) [ 1231.926229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1231.962202] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1241.893876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1242.825478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_dropped -r 1 [ 1242.865898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_dropped -r 1 [ 1242.929711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1242.950580] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1243.355975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.387814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.431447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.476696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.531562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.561303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.603745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.639940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.667400] LNet: 62801:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.693466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.727744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.802945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.876834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.947365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.012500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.065668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1244.968988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_error -r 1 [ 1245.056678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_error -r 1 [ 1245.132615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1245.190220] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1245.534113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.586332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.627278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.657899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.697570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.743786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.769809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.799542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.813982] LNet: 62910:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.831973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.855598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.878468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.905235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.926503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.973642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.030028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1246.792660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_timeout -r 1 [ 1246.840554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_timeout -r 1 [ 1246.895536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1246.913805] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1247.209884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.251431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.280896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.316440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.344264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.375227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.409438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.440838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.456657] LNet: 63019:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.474429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.501585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.551098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.593604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.636991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.671936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.708355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1247.976824] LNet: 63060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.983460] LNetError: 63060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.003569] LNet: Removed LNI 192.168.202.29@tcp [ 1248.491530] Key type .llcrypt unregistered [ 1248.497166] Key type ._llcrypt unregistered [ 1269.686439] Key type ._llcrypt registered [ 1269.688228] Key type .llcrypt registered [ 1271.328400] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:18:25 (1788844705) [ 1272.319334] Key type .llcrypt unregistered [ 1272.322750] Key type ._llcrypt unregistered [ 1279.635889] Key type ._llcrypt registered [ 1279.637407] Key type .llcrypt registered [ 1279.872632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1279.891822] alg: No test for adler32 (adler32-zlib) [ 1280.816736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1280.843829] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1289.145664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1298.896756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1298.960397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1298.990330] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1301.049106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_dropped -r 1 [ 1301.100308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e remote_dropped -r 1 [ 1301.147887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_dropped -r 1 [ 1301.208883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e remote_dropped -r 1 [ 1301.251471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e remote_dropped -r 1 [ 1301.304241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e remote_dropped -r 1 [ 1301.341513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e remote_dropped -r 1 [ 1301.377812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e remote_dropped -r 1 [ 1301.415171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1301.435655] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1301.441679] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.129@tcp [ 1301.694320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1301.749620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1301.845598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1301.901229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1301.956826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.021823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.087712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.133282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.174991] LNet: 64526:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.193684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.221179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.267078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.315720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1302.343390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.388240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.432939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1302.493786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1303.088389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_error -r 1 [ 1303.125982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e remote_error -r 1 [ 1303.161188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_error -r 1 [ 1303.218979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e remote_error -r 1 [ 1303.253982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e remote_error -r 1 [ 1303.289446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e remote_error -r 1 [ 1303.318210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e remote_error -r 1 [ 1303.350883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e remote_error -r 1 [ 1303.388702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1303.406825] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1303.411829] LNet: Skipped 2 previous similar messages [ 1303.638117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1303.657725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1303.677122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1303.719889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1303.751307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1303.799640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1303.831762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1303.876495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1303.905735] LNet: 64649:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1303.923451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1303.961454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1303.998812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.019424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.045660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.075506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1304.120326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1304.145849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.605752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_timeout -r 1 [ 1304.681908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e remote_timeout -r 1 [ 1304.744976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_timeout -r 1 [ 1304.776613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e remote_timeout -r 1 [ 1304.820856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e remote_timeout -r 1 [ 1304.859149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e remote_timeout -r 1 [ 1304.919782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e remote_timeout -r 1 [ 1304.959789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e remote_timeout -r 1 [ 1304.991260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1305.013052] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1305.243774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.289935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1305.329344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.359916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.390876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.421702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.451840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.485801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.500646] LNet: 64773:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.517961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.545878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.579700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.608753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.642296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.674822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.702807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1305.743863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1310.947756] LNet: 64851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.958147] LNetError: 64851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.964276] LNet: Removed LNI 192.168.202.29@tcp [ 1310.978335] LNet: Removed LNI 192.168.202.29@tcp1 [ 1311.504881] Key type .llcrypt unregistered [ 1311.507086] Key type ._llcrypt unregistered [ 1331.176742] Key type ._llcrypt registered [ 1331.178854] Key type .llcrypt registered [ 1332.600810] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:19:27 (1788844767) [ 1334.515921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1336.970739] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:19:31 (1788844771) [ 1338.136507] Key type .llcrypt unregistered [ 1338.140661] Key type ._llcrypt unregistered [ 1345.654537] Key type ._llcrypt registered [ 1345.656670] Key type .llcrypt registered [ 1345.871543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1345.892915] alg: No test for adler32 (adler32-zlib) [ 1346.937512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1346.993968] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1357.471859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1358.383578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e network_timeout -r 1 [ 1358.419910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e network_timeout -r 1 [ 1358.452497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1358.471359] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1358.793774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1358.849689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1358.897177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1358.934430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1358.958269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1358.986185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1359.065683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1359.126956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1359.163314] LNet: 66390:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1359.195221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1359.258894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.306652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.329533] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1359.332631] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1359.356299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1359.409161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1359.449529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1359.692264] LNet: 66429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.697179] LNetError: 66429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.720587] LNet: Removed LNI 192.168.202.29@tcp [ 1360.156694] Key type .llcrypt unregistered [ 1360.160916] Key type ._llcrypt unregistered [ 1382.961347] Key type ._llcrypt registered [ 1382.964760] Key type .llcrypt registered [ 1383.171836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1383.197725] alg: No test for adler32 (adler32-zlib) [ 1384.093984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1384.125466] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1392.206415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1402.949772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1403.005577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1403.050071] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1404.563022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e network_timeout -r 1 [ 1404.615860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e network_timeout -r 1 [ 1404.650168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e network_timeout -r 1 [ 1404.693538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e network_timeout -r 1 [ 1404.747903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e network_timeout -r 1 [ 1404.789517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e network_timeout -r 1 [ 1404.851276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e network_timeout -r 1 [ 1404.904104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e network_timeout -r 1 [ 1404.948899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1404.967400] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1405.207846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1405.239289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1405.276116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1405.314755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1405.353976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1405.393402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1405.418342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1405.440943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1405.461296] LNet: 67315:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1405.481175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1405.521907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1405.581398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1405.621861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1405.652931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1405.667569] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1405.677283] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1405.679423] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.129@tcp [ 1405.689469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1405.694706] LNet: Skipped 1 previous similar message [ 1407.718817] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1409.761259] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1411.611982] LNet: 67389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.619807] LNetError: 67389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.631365] LNet: Removed LNI 192.168.202.29@tcp [ 1411.648491] LNet: Removed LNI 192.168.202.29@tcp1 [ 1412.208162] Key type .llcrypt unregistered [ 1412.217840] Key type ._llcrypt unregistered [ 1431.157625] Key type ._llcrypt registered [ 1431.159653] Key type .llcrypt registered [ 1432.485928] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:21:07 (1788844867) [ 1432.986027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1433.018156] alg: No test for adler32 (adler32-zlib) [ 1433.996476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.039309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.059595] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1434.589725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1434.604485] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1435.601777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 1435.708620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1435.754458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1435.784479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1435.809171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1435.834790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1435.859975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.29@tcp [ 1435.878439] LNetError: 68487:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.29@tcp to recovery [ 1436.064559] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1436.065326] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1438.120359] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1438.320130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1442.209521] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1443.308313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1443.394890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1443.445328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1443.486326] LNet: 68594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.497493] LNetError: 68594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1443.510905] LNet: Removed LNI 192.168.202.29@tcp [ 1443.532057] LNet: Removed LNI 192.168.202.29@tcp1 [ 1443.557670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1443.650725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1443.689511] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1444.640788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1444.654663] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1445.400591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 1445.425654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1445.482568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1445.526962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1445.561679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1445.596641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1445.625881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1445.656588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.29@tcp [ 1445.676703] LNetError: 68867:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.29@tcp to recovery [ 1446.689887] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1449.248961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1452.832385] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1452.837128] LNet: Skipped 1 previous similar message [ 1457.339445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1457.450347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1457.514454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1466.594218] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:21:41 (1788844901) [ 1466.693385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.711886] LNet: 69578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.721688] LNetError: 69578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.726780] LNetError: 69578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1466.738326] LNet: Removed LNI 192.168.202.29@tcp [ 1466.757211] LNet: Removed LNI 192.168.202.29@tcp1 [ 1466.800672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1466.887648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1466.935626] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1468.139848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1468.200927] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1469.677775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 1469.730663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1469.768784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1469.805698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1469.839457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1469.885173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1469.920807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.29@tcp --health 0 [ 1470.043696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1470.944525] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.29@tcp [ 1470.945055] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1470.967840] LNet: Skipped 1 previous similar message [ 1484.980427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1485.168801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1485.215780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1485.252600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.29@tcp --health 500 [ 1485.511864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.543621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.664888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.705082] LNet: 70068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.713337] LNetError: 70068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.721553] LNetError: 70068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1492.737504] LNet: Removed LNI 192.168.202.29@tcp [ 1492.784590] LNet: Removed LNI 192.168.202.29@tcp1 [ 1492.865636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.961279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.006027] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1494.248434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1494.265834] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1495.465853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 1495.520721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1495.619156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1495.654216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1495.682650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1495.718960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1495.756343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1495.790829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.29@tcp --health 0 [ 1496.995382] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1497.002934] LNet: Skipped 3 previous similar messages [ 1503.723315] Lustre: DEBUG MARKER: sanity-lnet test_211: @@@@@@ FAIL: Unexpected ping count