[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 611611318 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003143] x2apic enabled [ 0.004000] Switched APIC routing to physical x2apic. [ 0.004027] kvm-guest: setup PV IPIs [ 0.005000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006000] pid_max: default: 32768 minimum: 301 [ 0.007141] LSM: Security Framework initializing [ 0.008083] Yama: becoming mindful. [ 0.009086] SELinux: Initializing. [ 0.010119] *** VALIDATE selinux *** [ 0.015000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.019748] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.020208] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.021138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.022168] *** VALIDATE tmpfs *** [ 0.023568] *** VALIDATE proc *** [ 0.025036] *** VALIDATE cgroup *** [ 0.026022] *** VALIDATE cgroup2 *** [ 0.028331] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030105] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032040] Spectre V2 : User space: Vulnerable [ 0.033018] Speculative Store Bypass: Vulnerable [ 0.036270] debug: unmapping init [mem 0xffffffff9a059000-0xffffffff9a060fff] [ 0.038328] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039901] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040034] ... version: 2 [ 0.041020] ... bit width: 48 [ 0.042018] ... generic registers: 4 [ 0.043025] ... value mask: 0000ffffffffffff [ 0.044028] ... max period: 00007fffffffffff [ 0.045026] ... fixed-purpose events: 3 [ 0.046030] ... event mask: 000000070000000f [ 0.047389] rcu: Hierarchical SRCU implementation. [ 0.051371] smp: Bringing up secondary CPUs ... [ 0.052741] x86: Booting SMP configuration: [ 0.053043] .... node #0, CPUs: #1 #2 #3 [ 0.083117] smp: Brought up 1 node, 4 CPUs [ 0.085019] smpboot: Max logical packages: 1 [ 0.086019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.184277] node 0 deferred pages initialised in 93ms [ 0.193044] devtmpfs: initialized [ 0.194840] x86/mm: Memory block size: 128MB [ 0.199431] gcov: version magic: 0x41383552 [ 0.205451] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.212052] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.218459] pinctrl core: initialized pinctrl subsystem [ 0.224024] [ 0.224705] ************************************************************* [ 0.230328] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.233015] ** ** [ 0.237016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.239016] ** ** [ 0.240016] ** This means that this kernel is built to expose internal ** [ 0.244020] ** IOMMU data structures, which may compromise security on ** [ 0.251021] ** your system. ** [ 0.262019] ** ** [ 0.267026] ** If you see this message and you are not debugging the ** [ 0.272016] ** kernel, report this immediately to your vendor! ** [ 0.276124] ** ** [ 0.281021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285022] ************************************************************* [ 0.297537] NET: Registered protocol family 16 [ 0.299991] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.308084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.310075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.320045] cpuidle: using governor menu [ 0.334684] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.344791] PCI: Using configuration type 1 for base access [ 0.349206] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.364150] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.365000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.378061] cryptd: max_cpu_qlen set to 1000 [ 0.384558] ACPI: Added _OSI(Module Device) [ 0.386026] ACPI: Added _OSI(Processor Device) [ 0.389023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.390000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.400011] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.422229] ACPI: Interpreter enabled [ 0.426099] ACPI: PM: (supports S0 S3 S4 S5) [ 0.429019] ACPI: Using IOAPIC for interrupt routing [ 0.432148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.435396] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.447610] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.450097] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.453029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.457186] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.463956] acpiphp: Slot [2] registered [ 0.465329] acpiphp: Slot [5] registered [ 0.467336] acpiphp: Slot [6] registered [ 0.469245] acpiphp: Slot [3] registered [ 0.471148] acpiphp: Slot [4] registered [ 0.473130] acpiphp: Slot [7] registered [ 0.474196] acpiphp: Slot [8] registered [ 0.476133] acpiphp: Slot [9] registered [ 0.478247] acpiphp: Slot [10] registered [ 0.480144] acpiphp: Slot [11] registered [ 0.482136] acpiphp: Slot [12] registered [ 0.483233] acpiphp: Slot [13] registered [ 0.485126] acpiphp: Slot [14] registered [ 0.487131] acpiphp: Slot [15] registered [ 0.489133] acpiphp: Slot [16] registered [ 0.490126] acpiphp: Slot [17] registered [ 0.493142] acpiphp: Slot [18] registered [ 0.494329] acpiphp: Slot [19] registered [ 0.496209] acpiphp: Slot [20] registered [ 0.498164] acpiphp: Slot [21] registered [ 0.500134] acpiphp: Slot [22] registered [ 0.502168] acpiphp: Slot [23] registered [ 0.504135] acpiphp: Slot [24] registered [ 0.505000] acpiphp: Slot [25] registered [ 0.505170] acpiphp: Slot [26] registered [ 0.506000] acpiphp: Slot [27] registered [ 0.509161] acpiphp: Slot [28] registered [ 0.512816] acpiphp: Slot [29] registered [ 0.516180] acpiphp: Slot [30] registered [ 0.518144] acpiphp: Slot [31] registered [ 0.520080] PCI host bridge to bus 0000:00 [ 0.523032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.526030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.530000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.535034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.538031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.541035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.543220] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.549287] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.554085] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.565021] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.571059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.577029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.583036] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.590023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.596000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.606980] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.615059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.619101] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 11718 usecs [ 0.625720] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.632828] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.649020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.654571] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.668407] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.678023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.684017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.701023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.708000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.714024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.720023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.731024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.742000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.744111] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.747351] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.748446] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.750198] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.757254] iommu: Default domain type: Passthrough [ 0.760715] SCSI subsystem initialized [ 0.763212] ACPI: bus type USB registered [ 0.766157] usbcore: registered new interface driver usbfs [ 0.768102] usbcore: registered new interface driver hub [ 0.770176] usbcore: registered new device driver usb [ 0.772206] pps_core: LinuxPPS API ver. 1 registered [ 0.775021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.778122] PTP clock support registered [ 0.781110] EDAC MC: Ver: 3.0.0 [ 0.783193] PCI: Using ACPI for IRQ routing [ 0.786751] NetLabel: Initializing [ 0.788018] NetLabel: domain hash size = 128 [ 0.790024] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.793300] NetLabel: unlabeled traffic allowed by default [ 0.797939] vgaarb: loaded [ 0.800388] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.802013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.811122] clocksource: Switched to clocksource kvm-clock [ 1.022205] VFS: Disk quotas dquot_6.6.0 [ 1.031145] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.036456] *** VALIDATE ramfs *** [ 1.044011] *** VALIDATE hugetlbfs *** [ 1.045448] pnp: PnP ACPI init [ 1.055945] pnp: PnP ACPI: found 6 devices [ 1.095919] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.099103] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.107291] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.111896] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.115903] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.119339] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.123887] NET: Registered protocol family 2 [ 1.127876] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.138876] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.146943] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.154693] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.158937] TCP: Hash tables configured (established 65536 bind 65536) [ 1.162726] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.166407] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.172996] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.186520] NET: Registered protocol family 1 [ 1.207059] RPC: Registered named UNIX socket transport module. [ 1.211819] RPC: Registered udp transport module. [ 1.215686] RPC: Registered tcp transport module. [ 1.217805] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.223392] NET: Registered protocol family 44 [ 1.225598] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.230625] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.232957] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.236507] PCI: CLS 0 bytes, default 64 [ 1.239206] Unpacking initramfs... [ 3.931906] debug: unmapping init [mem 0xffff9f66bcc64000-0xffff9f66bffcffff] [ 3.941690] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.951443] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.956150] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.660241] Initialise system trusted keyrings [ 4.662038] Key type blacklist registered [ 4.663807] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.673588] zbud: loaded [ 4.678734] *** VALIDATE nfs *** [ 4.680068] *** VALIDATE nfs4 *** [ 4.682338] pstore: using deflate compression [ 4.687190] Platform Keyring initialized [ 4.826896] NET: Registered protocol family 38 [ 4.829377] Key type asymmetric registered [ 4.832054] Asymmetric key parser 'x509' registered [ 4.833765] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.842537] io scheduler mq-deadline registered [ 4.846185] io scheduler kyber registered [ 4.850468] io scheduler bfq registered [ 4.852732] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.856625] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.860709] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.868573] ACPI: Power Button [PWRF] [ 4.875623] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.883911] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.910620] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.960253] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.003886] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.022622] Non-volatile memory driver v1.3 [ 5.025504] Linux agpgart interface v0.103 [ 5.080184] virtio_blk virtio1: [vda] 146024 512-byte logical blocks (74.8 MB/71.3 MiB) [ 5.085693] vda: detected capacity change from 0 to 74764288 [ 5.117048] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.135804] vdb: detected capacity change from 0 to 1073741824 [ 5.159526] libphy: Fixed MDIO Bus: probed [ 5.172335] usbcore: registered new interface driver usbserial_generic [ 5.177089] usbserial: USB Serial support registered for generic [ 5.182663] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.188648] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.191144] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.194327] mousedev: PS/2 mouse device common for all mice [ 5.199354] rtc_cmos 00:05: RTC can wake from S4 [ 5.202947] rtc_cmos 00:05: registered as rtc0 [ 5.206295] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.210832] intel_pstate: CPU model not supported [ 5.215148] hid: raw HID events driver (C) Jiri Kosina [ 5.218822] usbcore: registered new interface driver usbhid [ 5.221724] usbhid: USB HID core driver [ 5.226928] drop_monitor: Initializing network drop monitor service [ 5.228447] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.230641] Initializing XFRM netlink socket [ 5.238522] NET: Registered protocol family 10 [ 5.245616] Segment Routing with IPv6 [ 5.246874] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.248858] NET: Registered protocol family 17 [ 5.258768] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.259659] mpls_gso: MPLS GSO support [ 5.270859] RAS: Correctable Errors collector initialized. [ 5.275282] AVX version of gcm_enc/dec engaged. [ 5.277545] AES CTR mode by8 optimization enabled [ 5.515458] sched_clock: Marking stable (5515418441, 0)->(7667509861, -2152091420) [ 5.523719] registered taskstats version 1 [ 5.526953] Loading compiled-in X.509 certificates [ 5.530224] zswap: loaded using pool lzo/zbud [ 5.605602] Key type big_key registered [ 5.623363] Key type encrypted registered [ 5.625227] ima: No TPM chip found, activating TPM-bypass! [ 5.636832] ima: Allocated hash algorithm: sha1 [ 5.639547] ima: No architecture policies found [ 5.647403] evm: Initialising EVM extended attributes: [ 5.649243] evm: security.selinux [ 5.650374] evm: security.ima [ 5.652239] evm: security.capability [ 5.653424] evm: HMAC attrs: 0x1 [ 5.662537] rtc_cmos 00:05: setting system clock to 2026-09-02 00:37:15 UTC (1788309435) [ 5.689278] debug: unmapping init [mem 0xffffffff9b003000-0xffffffff9b1fffff] [ 5.697461] debug: unmapping init [mem 0xffffffff99d82000-0xffffffff9a058fff] [ 5.705202] Write protecting the kernel read-only data: 28672k [ 5.708796] debug: unmapping init [mem 0xffffffff98403000-0xffffffff985fffff] [ 5.717925] debug: unmapping init [mem 0xffffffff98d14000-0xffffffff98dfffff] [ 5.848304] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.865850] systemd[1]: Detected virtualization kvm. [ 5.869164] systemd[1]: Detected architecture x86-64. [ 5.871608] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.949169] systemd[1]: No hostname configured. [ 5.954747] systemd[1]: Set hostname to . [ 5.960322] random: systemd: uninitialized urandom read (16 bytes read) [ 5.963738] systemd[1]: Initializing machine ID from random generator. [ 6.100386] random: ln: uninitialized urandom read (6 bytes read) [ 6.269607] random: systemd: uninitialized urandom read (16 bytes read) [ 6.276705] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.283885] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.292475] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.398164] device-mapper: uevent: version 1.0.3 [ 7.400752] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.819156] virtio_net virtio0 ens2: renamed from eth0 [ 8.851590] random: fast init done [ 8.957240] scsi host0: ata_piix [ 8.978753] scsi host1: ata_piix [ 8.980477] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.988622] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 14.172462] random: crng init done [ 14.176084] random: 7 urandom warning(s) missed due to ratelimiting [ 14.219076] dracut-initqueue[577]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.167238] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.290830] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.002353] SELinux: Disabled at runtime. [ 19.085645] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.101764] systemd[1]: Detected virtualization kvm. [ 19.105645] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.324217] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.336653] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.349931] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.358047] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.362225] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.380262] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.398189] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 20.672701] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 22.031764] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.878086] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.936380] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.327401] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.460916] EDAC sbridge: Ver: 1.1.2 [ 25.497092] Key type dns_resolver registered [ 26.220607] NFS: Registering the id_resolver key type [ 26.225774] Key type id_resolver registered [ 26.232494] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg142-client login: [ 62.619020] hrtimer: interrupt took 5010560 ns [ 78.658904] libcfs: loading out-of-tree module taints kernel. [ 78.796537] Key type ._llcrypt registered [ 78.803474] Key type .llcrypt registered [ 79.224450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 79.242416] alg: No test for adler32 (adler32-zlib) [ 80.626162] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 81.293811] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 83.015164] Key type lgssc registered [ 84.673625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 200.424796] Lustre: Mounted lustre-client - version 2.17.57_45_g07d9be3 [ 205.593378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 223.092666] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing check_logdir /tmp/testlogs/ [ 226.271336] Lustre: lustre-OST0000-osc-ffff9f6702be7800: disconnect after 24s idle [ 228.593779] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing yml_node [ 233.297956] Lustre: DEBUG MARKER: Client: 2.17.57.45 [ 235.601969] Lustre: DEBUG MARKER: MDS: 2.17.57.45 [ 238.066506] Lustre: DEBUG MARKER: OSS: 2.17.57.45 [ 239.757631] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 1 20:41:08 EDT 2026 [ 255.857249] Lustre: Unmounted lustre-client [ 276.328368] Key type lgssc unregistered [ 276.507118] LNet: 8453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.519826] LNetError: 8453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 276.536365] LNet: Removed LNI 192.168.201.42@tcp [ 277.094160] Key type .llcrypt unregistered [ 277.095195] Key type ._llcrypt unregistered [ 304.481447] Key type ._llcrypt registered [ 304.486729] Key type .llcrypt registered [ 304.784524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 304.815650] alg: No test for adler32 (adler32-zlib) [ 305.792744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 305.902070] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 305.952494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 310.549496] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 310.954228] LNet: 9341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.963819] LNetError: 9341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 310.980685] LNet: Removed LNI 192.168.201.42@tcp [ 311.811365] Key type .llcrypt unregistered [ 311.812954] Key type ._llcrypt unregistered [ 318.715901] Key type ._llcrypt registered [ 318.718651] Key type .llcrypt registered [ 320.097188] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:42:28 (1788309748) [ 320.411495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 320.423974] alg: No test for adler32 (adler32-zlib) [ 321.374967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.441729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.455705] LNet: 9756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.469759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.504134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 321.516789] LNet: 9763:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 327.630176] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:42:36 (1788309756) [ 327.836584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.869139] LNet: 10340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.894072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.989841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 333.550546] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:42:42 (1788309762) [ 333.687640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.719871] LNet: 10922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.765278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.850729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 339.706224] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:42:48 (1788309768) [ 339.985756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.016359] LNet: 11512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.061565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.132746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 345.982904] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:42:54 (1788309774) [ 346.233792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.258789] LNet: 12097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.274601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.336877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 346.781226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 352.572051] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:43:00 (1788309780) [ 352.762579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.820863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.919176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 353.559409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 354.135506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 360.934342] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:43:09 (1788309789) [ 361.725674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.745413] LNet: 13284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.753435] LNet: 13284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 361.783791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.845473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 369.776167] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:43:17 (1788309797) [ 370.004348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.105782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.223867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 370.284214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 370.860047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 370.914569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 371.477541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 371.553950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 372.333242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 372.415306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 372.776779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 372.827542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 373.277861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 373.313377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 373.634685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 373.676779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 374.151531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 374.219848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 374.575248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 374.603761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 374.941370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 375.052563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 375.388820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 375.405687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 375.862945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 375.924443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.339256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 376.355526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 383.350875] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:43:31 (1788309811) [ 383.723410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.747194] LNet: 14552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.749817] LNet: 14552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 383.783213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.838428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 383.871557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 391.685070] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:43:39 (1788309819) [ 391.884532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.931136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.154737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 392.252451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 399.932833] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:43:47 (1788309827) [ 400.112241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.246355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.555295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 400.663647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 408.807281] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:43:57 (1788309837) [ 408.904906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.937471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.977877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 409.000277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 415.614668] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:44:04 (1788309844) [ 415.678059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.711626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.769910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 415.787173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 421.463061] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:44:09 (1788309849) [ 421.671312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.696954] LNet: 17482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.701562] LNet: 17482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 421.734620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.793619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 421.831663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 428.263518] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:44:16 (1788309856) [ 428.343176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.383465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.448185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.687661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 433.992166] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:44:22 (1788309862) [ 434.084185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.112413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.344371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.527373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.775254] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:44:28 (1788309868) [ 439.831935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.861850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.951944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.201565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 445.864840] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:44:34 (1788309874) [ 446.106633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.151825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.698637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.933037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.886815] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:44:41 (1788309881) [ 452.989901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.036189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.097711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.136799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 458.515613] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:44:47 (1788309887) [ 458.628195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.678688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.761735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.803451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.808773] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:44:53 (1788309893) [ 464.914046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.972754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.065314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 470.845889] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:44:59 (1788309899) [ 470.910634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.954055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.010518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 476.200496] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:45:04 (1788309904) [ 476.311235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.385196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.465512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 476.926937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 482.749316] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:45:11 (1788309911) [ 483.012873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.073799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.133409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 483.162713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 488.481539] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:45:17 (1788309917) [ 488.591537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.622221] LNet: 24094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 488.631491] LNet: 24094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 488.656062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.729588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 488.762447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 494.193466] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:45:22 (1788309922) [ 494.439250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.479920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.549905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 494.593498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 500.561994] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:45:28 (1788309928) [ 500.817632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.885613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.981393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 501.027931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 507.663464] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:45:35 (1788309935) [ 507.861474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.926564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.958942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 508.013271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 508.034395] LNetError: 25866:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 508.091101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 508.150674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 514.154306] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:45:42 (1788309942) [ 514.329247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.358340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.443443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.95KR1iQ7U5 [ 514.532659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.95KR1iQ7U5 [ 514.576887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.95KR1iQ7U5 [ 519.530838] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:45:48 (1788309948) [ 519.642680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.683277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.762517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 519.802440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 519.862312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 525.220254] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 20:45:53 (1788309953) [ 525.581922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.632605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.751841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 525.771329] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 526.979335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 528.167505] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 530.095164] Key type lgssc registered [ 531.185323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 534.214529] LNetError: 27627:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.42@tcp99 has no available nets [ 536.591900] Key type lgssc unregistered [ 536.920436] LNetError: 28738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 536.945855] LNet: Removed LNI 192.168.201.42@tcp [ 537.503583] Key type .llcrypt unregistered [ 537.505951] Key type ._llcrypt unregistered [ 543.552368] Key type ._llcrypt registered [ 543.555813] Key type .llcrypt registered [ 544.897948] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:46:13 (1788309973) [ 545.484307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 545.512149] alg: No test for adler32 (adler32-zlib) [ 546.556401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.629671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 549.688637] LNet: Message forwarding will be enabled in 2s [ 552.800915] LNet: Message forwarding enabled [ 553.182825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 553.204769] LNet: Message forwarding will stop in 2s to 6s [ 557.644660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 558.585382] LNet: Message forwarding disabled [ 560.077682] LNet: Message forwarding will be enabled in 2s [ 563.743132] LNet: Message forwarding enabled [ 566.462239] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:46:34 (1788309994) [ 566.641686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.663849] LNet: 30141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.283940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.365951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 572.950434] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:46:41 (1788310001) [ 573.270591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.292516] LNet: 30727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.305716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.336354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 573.351387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 573.364745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 573.379893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 573.394783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 573.409417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 573.421661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 573.440452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 573.462927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 573.483659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 573.503899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 573.525895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 573.558924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 573.609862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 573.651901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 573.684909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 573.712928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 573.746123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 573.801360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 573.853953] LNetError: 30771:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 581.290350] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:46:49 (1788310009) [ 581.624554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.654133] LNet: 31342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.683409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.814566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 589.393447] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:46:57 (1788310017) [ 591.285820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg142-server [ 593.006639] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:47:01 (1788310021) [ 593.363531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.383772] LNet: 32273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.401531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.454310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 593.469655] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 593.953879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[43-47]@tcp [ 593.969425] LNet: 32406:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 593.982863] LNetError: 32283:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.44@tcp has gone from up to down [ 593.997409] LNetError: 32277:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.45@tcp is being used as a gateway but routing feature is not turned on [ 595.073574] LNetError: 32277:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.45@tcp is being used as a gateway but routing feature is not turned on [ 595.087232] LNetError: 32277:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 596.082769] LNetError: 32277:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.46@tcp is being used as a gateway but routing feature is not turned on [ 596.103473] LNetError: 32277:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 598.120642] LNetError: 32277:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.43@tcp is being used as a gateway but routing feature is not turned on [ 598.137919] LNetError: 32277:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 600.317551] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:47:09 (1788310029) [ 602.226282] LNetError: 32276:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.201.44@tcp [ 602.243157] LNetError: 32277:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.43@tcp is being used as a gateway but routing feature is not turned on [ 602.259898] LNetError: 32277:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 602.308387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg142-server [ 603.839861] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:47:12 (1788310032) [ 604.065491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.089951] LNet: 33342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.098389] LNetError: 33342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.123793] LNet: Removed LNI 192.168.201.42@tcp [ 604.190179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.253904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 604.295338] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 604.873842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[43-48/2]@tcp [ 604.893683] LNet: 33459:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 604.900630] LNet: 33459:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 604.921994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[43-48/2]@tcp [ 604.925731] LNetError: 33346:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.45@tcp has gone from up to down [ 604.939708] LNetError: 33346:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 611.257228] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:47:19 (1788310039) [ 611.407847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.434550] LNetError: 34047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.470551] LNet: Removed LNI 192.168.201.42@tcp [ 611.532716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.643223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 611.678142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.718924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.805474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 611.857513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.921228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.000989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 612.093806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.112310] LNet: 34090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.117473] LNet: 34090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 612.140888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.182655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 612.237823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.265644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.307728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 612.382217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.415745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.478766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 612.558264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.603254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.629424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 612.699917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.736179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.799791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 612.874434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.913366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.952218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 613.010225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.037571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.078343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 613.115437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.129985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.162096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 613.209646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.232585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.270846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 619.283837] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:47:27 (1788310047) [ 620.947236] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg142-server [ 622.892318] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:47:31 (1788310051) [ 625.300149] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg142-server [ 627.130985] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:47:35 (1788310055) [ 628.233692] Key type .llcrypt unregistered [ 628.235359] Key type ._llcrypt unregistered [ 635.023116] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 635.322648] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 635.332707] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 635.342561] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 637.872825] Key type ._llcrypt registered [ 637.874866] Key type .llcrypt registered [ 638.089815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 638.132523] alg: No test for adler32 (adler32-zlib) [ 639.000354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.081585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 639.099438] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 639.551945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 639.571405] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 641.442355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 641.491304] LNet: Removed LNI 10.1.2.3@tcp [ 641.688393] LNet: 36179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.693199] LNetError: 36179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.705645] LNet: Removed LNI 192.168.201.42@tcp [ 642.402784] Key type .llcrypt unregistered [ 642.407676] Key type ._llcrypt unregistered [ 652.973188] Key type ._llcrypt registered [ 652.975368] Key type .llcrypt registered [ 654.061622] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:48:02 (1788310082) [ 654.541424] Key type .llcrypt unregistered [ 654.543986] Key type ._llcrypt unregistered [ 659.652965] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 659.803148] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 659.811421] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 659.815552] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 662.386867] Key type ._llcrypt registered [ 662.390828] Key type .llcrypt registered [ 662.567116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 662.587641] alg: No test for adler32 (adler32-zlib) [ 663.592326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.656872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 663.685262] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 664.475394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 664.499110] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 666.205890] LNet: 37676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.221853] LNetError: 37676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 666.234777] LNet: Removed LNI 10.1.2.3@tcp [ 666.246105] LNet: Removed LNI 192.168.201.42@tcp [ 666.725664] Key type .llcrypt unregistered [ 666.727457] Key type ._llcrypt unregistered [ 677.528145] Key type ._llcrypt registered [ 677.529355] Key type .llcrypt registered [ 678.636077] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:48:27 (1788310107) [ 679.346267] Key type .llcrypt unregistered [ 679.348834] Key type ._llcrypt unregistered [ 684.608441] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 684.640051] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 684.642451] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 684.644977] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 687.384577] Key type ._llcrypt registered [ 687.387888] Key type .llcrypt registered [ 687.607466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 687.640462] alg: No test for adler32 (adler32-zlib) [ 688.615328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.675168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 688.710118] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 690.159529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 690.175937] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 691.902628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 691.921710] LNet: Removed LNI 10.1.2.3@tcp [ 691.934927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 691.963460] LNet: Removed LNI 10.1.2.31@tcp [ 691.982588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 692.012982] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 692.819326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 692.838486] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 693.714973] LNet: 39607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.719254] LNetError: 39607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 693.724674] LNet: Removed LNI 10.1.2.31@tcp [ 693.731497] LNet: Removed LNI 10.1.2.3@tcp [ 694.233624] Key type .llcrypt unregistered [ 694.235915] Key type ._llcrypt unregistered [ 706.161584] Key type ._llcrypt registered [ 706.163181] Key type .llcrypt registered [ 707.710496] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:48:55 (1788310135) [ 708.145169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 708.159711] alg: No test for adler32 (adler32-zlib) [ 709.070909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.158827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 709.208308] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 709.886294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.914271] LNet: 40565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.919662] LNetError: 40565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.932967] LNet: Removed LNI 192.168.201.42@tcp [ 709.952391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.999880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.42@tcp [ 710.023345] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 710.697938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.42@tcp [ 710.728436] LNet: Removed LNI 192.168.201.42@tcp [ 710.782912] LNet: 40700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.385819] Key type .llcrypt unregistered [ 711.388991] Key type ._llcrypt unregistered [ 721.654362] Key type ._llcrypt registered [ 721.656525] Key type .llcrypt registered [ 722.769321] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:49:11 (1788310151) [ 724.641792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg142-server [ 726.566686] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:49:14 (1788310154) [ 727.212519] Key type .llcrypt unregistered [ 727.217720] Key type ._llcrypt unregistered [ 736.822179] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 736.985922] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 736.991543] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 737.006147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 739.520282] Key type ._llcrypt registered [ 739.522567] Key type .llcrypt registered [ 739.765426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 739.791094] alg: No test for adler32 (adler32-zlib) [ 740.810617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.872811] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 741.966157] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 744.809305] LNet: 42544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.819921] LNetError: 42544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.825914] LNet: Removed LNI 192.168.201.42@tcp1 [ 744.843072] LNet: Removed LNI 10.1.2.3@tcp2 [ 746.090578] Key type .llcrypt unregistered [ 746.098694] Key type ._llcrypt unregistered [ 760.735248] Key type ._llcrypt registered [ 760.736685] Key type .llcrypt registered [ 762.318871] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:49:50 (1788310190) [ 762.790700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 762.895037] alg: No test for adler32 (adler32-zlib) [ 763.827945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.945970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.998621] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 764.926797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.937237] LNet: 43519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.940565] LNetError: 43519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.949793] LNet: Removed LNI 192.168.201.42@tcp [ 764.965336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.997798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 765.014726] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 771.539988] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:50:00 (1788310200) [ 771.984490] LNet: 44110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.989079] LNetError: 44110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.002515] LNet: Removed LNI 192.168.201.42@tcp [ 772.699724] Key type .llcrypt unregistered [ 772.701465] Key type ._llcrypt unregistered [ 779.191730] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.428932] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.434724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.438058] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 782.033643] Key type ._llcrypt registered [ 782.035957] Key type .llcrypt registered [ 782.273385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 782.287152] alg: No test for adler32 (adler32-zlib) [ 783.190486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.238899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 783.257216] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 783.853791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 783.891795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.462263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.473527] LNet: 44857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.479492] LNetError: 44857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 785.483203] LNet: Removed LNI 10.1.2.3@tcp [ 785.494736] LNet: Removed LNI 192.168.201.42@tcp [ 785.514886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.563873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 785.588048] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 785.595926] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.976248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.992459] LNet: 44884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.996950] LNetError: 44884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 786.000860] LNetError: 44884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 786.006279] LNet: Removed LNI 10.1.2.3@tcp [ 786.019995] LNet: Removed LNI 192.168.201.42@tcp [ 786.034077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.093209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 786.124417] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 786.127943] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 786.427842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 786.437454] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 787.363346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.384271] LNet: 45046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.393895] LNetError: 45046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 787.400944] LNetError: 45046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 787.406585] LNet: Removed LNI 10.1.2.3@tcp [ 787.408737] LNet: Removed LNI 192.168.201.42@tcp [ 787.420576] LNet: Removed LNI 192.168.201.42@tcp2 [ 787.436221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.491970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 787.508905] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 787.511750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 787.514612] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 793.573268] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:50:22 (1788310222) [ 793.693665] LNet: 45641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.701996] LNetError: 45641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 793.707842] LNetError: 45641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 793.722548] LNet: Removed LNI 10.1.2.3@tcp [ 793.728491] LNet: Removed LNI 192.168.201.42@tcp [ 793.756761] LNet: Removed LNI 192.168.201.42@tcp2 [ 794.304340] Key type .llcrypt unregistered [ 794.308366] Key type ._llcrypt unregistered [ 799.907812] Key type ._llcrypt registered [ 799.910224] Key type .llcrypt registered [ 800.117662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.130787] alg: No test for adler32 (adler32-zlib) [ 801.046347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.114077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.162935] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 801.831356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.844798] LNet: 46042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.847895] LNetError: 46042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.871947] LNet: Removed LNI 192.168.201.42@tcp [ 801.893379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.939250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 801.973852] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 802.229929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.254909] LNet: Removed LNI 192.168.201.42@tcp [ 802.270607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.315509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 802.327437] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 802.641869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.659190] LNet: 46102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.663725] LNet: 46102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 802.667606] LNetError: 46102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.673774] LNetError: 46102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 802.685931] LNet: Removed LNI 192.168.201.42@tcp [ 802.701118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.741896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 802.765192] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 803.295484] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.474706] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.482437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.485916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.889848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.920585] LNet: 46180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.928429] LNetError: 46180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.956056] LNet: Removed LNI 192.168.201.42@tcp [ 805.984438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.053845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 806.082064] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 812.038833] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:50:40 (1788310240) [ 812.158642] LNet: 46770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.162576] LNetError: 46770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.185647] LNet: Removed LNI 192.168.201.42@tcp [ 812.772974] Key type .llcrypt unregistered [ 812.776866] Key type ._llcrypt unregistered [ 818.804289] Key type ._llcrypt registered [ 818.807713] Key type .llcrypt registered [ 819.040299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 819.050483] alg: No test for adler32 (adler32-zlib) [ 820.000894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.041811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.062676] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 820.811317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.832973] LNet: 47180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.838644] LNetError: 47180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.846772] LNet: Removed LNI 192.168.201.42@tcp [ 820.858100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.893374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 820.913674] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 826.626410] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:50:55 (1788310255) [ 826.730669] LNet: 47766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.739188] LNetError: 47766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.756259] LNet: Removed LNI 192.168.201.42@tcp [ 827.355448] Key type .llcrypt unregistered [ 827.357690] Key type ._llcrypt unregistered [ 833.168949] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 833.457498] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 833.461609] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 833.464187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 836.127600] Key type ._llcrypt registered [ 836.131624] Key type .llcrypt registered [ 836.284197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 836.299934] alg: No test for adler32 (adler32-zlib) [ 837.272469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.338173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 837.361549] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 838.157674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.171202] LNet: 48238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.175976] LNetError: 48238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.190867] LNet: Removed LNI 192.168.201.42@tcp [ 838.206314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.290614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 838.316265] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 845.132758] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:51:13 (1788310273) [ 845.433685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.451841] LNet: 48836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.460803] LNetError: 48836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.477277] LNet: Removed LNI 192.168.201.42@tcp [ 845.495964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.653353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 851.174069] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:51:19 (1788310279) [ 851.391824] LNet: 49417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.907533] Key type .llcrypt unregistered [ 851.909156] Key type ._llcrypt unregistered [ 857.093870] Key type ._llcrypt registered [ 857.097658] Key type .llcrypt registered [ 857.350219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 857.391491] alg: No test for adler32 (adler32-zlib) [ 858.302145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.356067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 858.379201] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 859.035610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.052956] LNet: 49819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.056742] LNetError: 49819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.070765] LNet: Removed LNI 192.168.201.42@tcp [ 859.083383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.130856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 859.167972] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 865.131280] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:51:33 (1788310293) [ 865.212070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.241848] LNet: 50405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.256301] LNetError: 50405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.281991] LNet: Removed LNI 192.168.201.42@tcp [ 865.327624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.384974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 871.083265] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:51:39 (1788310299) [ 872.422736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 873.857604] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:51:42 (1788310302) [ 875.269609] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 876.780566] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:51:45 (1788310305) [ 878.058757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 879.514838] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:51:48 (1788310308) [ 880.628871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 882.439300] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:51:50 (1788310310) [ 883.632950] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 884.956404] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 20:51:53 (1788310313) [ 885.154331] LNet: 52463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.660652] Key type .llcrypt unregistered [ 885.662200] Key type ._llcrypt unregistered [ 890.935660] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.099153] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.102317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.105821] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 893.636681] Key type ._llcrypt registered [ 893.645789] Key type .llcrypt registered [ 893.792528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.807610] alg: No test for adler32 (adler32-zlib) [ 894.684791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.730626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 894.744230] LNet: Added LNI 192.168.201.42@tcp [8/512/0/180] [ 894.748216] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 894.749592] LNet: 52786:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 894.814699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.842915] LNet: 52805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.850880] LNetError: 52805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 894.855209] LNet: Removed LNI 10.1.2.3@tcp [ 894.857474] LNetError: 52805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.864733] LNet: Removed LNI 192.168.201.42@tcp [ 894.877279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.917815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 894.932797] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 894.975805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.989900] LNet: 52830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.993781] LNetError: 52830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.008988] LNet: Removed LNI 192.168.201.42@tcp [ 895.027842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.098202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 895.115469] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 895.117519] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.120428] LNet: 52838:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 895.227441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.251533] LNet: 52856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.257572] LNetError: 52856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.263466] LNet: Removed LNI 10.1.2.3@tcp [ 895.268399] LNetError: 52856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.280247] LNet: Removed LNI 192.168.201.42@tcp [ 895.295972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.351256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 895.375517] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 895.382609] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.386404] LNet: 52864:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 895.556865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.569273] LNet: 52890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.572933] LNetError: 52890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.575381] LNet: Removed LNI 10.1.2.3@tcp [ 895.578452] LNetError: 52890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.589426] LNet: Removed LNI 192.168.201.42@tcp [ 895.599979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.644203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 895.661113] LNet: Added LNI 192.168.201.42@tcp [8/256/0/182] [ 895.666461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 895.765779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.791496] LNet: 52918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.795203] LNetError: 52918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.804347] LNet: Removed LNI 10.1.2.3@tcp [ 895.808241] LNetError: 52918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.830761] LNet: Removed LNI 192.168.201.42@tcp [ 895.878593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.952692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 895.992103] LNet: Added LNI 192.168.201.42@tcp [8/256/0/182] [ 896.002129] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 896.005264] LNet: 52926:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 896.072954] LNet: Removed LNI 10.1.2.3@tcp [ 896.080227] LNet: Removed LNI 192.168.201.42@tcp [ 896.672766] Key type .llcrypt unregistered [ 896.675515] Key type ._llcrypt unregistered [ 906.887408] Key type ._llcrypt registered [ 906.889453] Key type .llcrypt registered [ 908.095843] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 20:52:16 (1788310336) [ 909.377874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 910.887348] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:52:19 (1788310339) [ 911.265781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 911.273957] alg: No test for adler32 (adler32-zlib) [ 912.137330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.178895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 912.197554] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 912.767788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.42 [ 912.789969] LNet: 54195:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 912.799069] LNet: Removed LNI 192.168.201.42@tcp [ 918.067680] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:52:26 (1788310346) [ 918.261369] LNet: 54764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.289177] LNetError: 54764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.325890] LNet: Removed LNI 192.168.201.42@tcp [ 918.969849] Key type .llcrypt unregistered [ 918.973370] Key type ._llcrypt unregistered [ 925.304207] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 925.566656] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 928.446231] Key type ._llcrypt registered [ 928.450302] Key type .llcrypt registered [ 928.627054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.639753] alg: No test for adler32 (adler32-zlib) [ 929.870141] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 935.973645] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:52:44 (1788310364) [ 936.083803] LNet: 55677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.091860] LNetError: 55677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 936.105230] LNet: Removed LNI 10.1.2.3@tcp [ 936.539678] Key type .llcrypt unregistered [ 936.541353] Key type ._llcrypt unregistered [ 942.351655] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 942.588715] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 942.599493] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 942.611323] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 945.395552] Key type ._llcrypt registered [ 945.400457] Key type .llcrypt registered [ 945.574659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.590967] alg: No test for adler32 (adler32-zlib) [ 946.504349] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 951.272937] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:53:00 (1788310380) [ 952.323878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 953.752831] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:53:02 (1788310382) [ 954.103320] LNet: 56871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.109703] LNetError: 56871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 954.128732] LNet: Removed LNI 10.1.2.3@tcp [ 954.670501] Key type .llcrypt unregistered [ 954.675214] Key type ._llcrypt unregistered [ 959.442642] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 959.712557] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 959.715708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 959.719667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 962.518731] Key type ._llcrypt registered [ 962.521019] Key type .llcrypt registered [ 962.741676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 962.757706] alg: No test for adler32 (adler32-zlib) [ 963.674686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.736477] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 972.034595] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:53:20 (1788310400) [ 972.558154] LNet: 58103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.572037] LNetError: 58103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 972.611146] LNet: Removed LNI 10.1.2.3@tcp [ 973.527580] Key type .llcrypt unregistered [ 973.530199] Key type ._llcrypt unregistered [ 980.463559] Key type ._llcrypt registered [ 980.465609] Key type .llcrypt registered [ 980.673877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 980.686881] alg: No test for adler32 (adler32-zlib) [ 981.595358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 981.618334] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 989.267074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 989.917518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_interrupt -r 1 [ 989.947840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_interrupt -r 1 [ 989.973462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 989.987679] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 990.188670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 990.227487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 990.258820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 990.298928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 990.322642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 990.346869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 990.370842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 990.404182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 990.419284] LNet: 58545:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 990.433751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 990.475106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 990.532119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.588449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 990.638371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 990.687732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 990.736827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 991.289138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_dropped -r 1 [ 991.328415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_dropped -r 1 [ 991.359843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 991.374554] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 991.666766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 991.720725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 991.756289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 991.802532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 991.846466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 991.893509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 991.906405] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 991.934381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 991.970979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.011882] LNet: 58654:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 992.040395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 992.096397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 992.147381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 992.203760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 992.265098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 992.308326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 992.334483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 992.954765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_aborted -r 1 [ 993.011633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_aborted -r 1 [ 993.038837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 993.050325] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 993.371856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 993.426404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 993.479757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 993.570514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 993.662180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 993.692198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 993.749226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 993.795465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 993.857850] LNet: 58764:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 993.880375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 993.925305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 993.971520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 994.017821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 994.061241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 994.112376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 994.158362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 994.806324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_no_route -r 1 [ 994.842599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_no_route -r 1 [ 994.876098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 995.096524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 995.134901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 995.161662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 995.181293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 995.206582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 995.231911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 995.260588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 995.297711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 995.341254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 995.385815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 995.436722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 995.485746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 995.531457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 995.564686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 995.621284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 996.110623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_timeout -r 1 [ 996.143929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_timeout -r 1 [ 996.196687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 996.213156] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 996.231907] LNet: Skipped 2 previous similar messages [ 996.422832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 996.454882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 996.512744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 996.541798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 996.571995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 996.601522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 996.637844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 996.688555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 996.713715] LNet: 58982:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 996.720708] LNet: 58982:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 996.749319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 996.783527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 996.808579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 996.825260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 996.860668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 996.925656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 996.964063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 997.378457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_error -r 1 [ 997.409193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_error -r 1 [ 997.429818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 997.551292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 997.567898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 997.596268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 997.622845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 997.650450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 997.684117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 997.708468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 997.733373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 997.757340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 997.774662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 997.792845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 997.821760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 997.867830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 997.924561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 997.961377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 998.207596] LNet: 59136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.218056] LNetError: 59136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.272956] LNet: Removed LNI 192.168.201.42@tcp [ 999.862838] Key type .llcrypt unregistered [ 999.866241] Key type ._llcrypt unregistered [ 1018.461650] Key type ._llcrypt registered [ 1018.462931] Key type .llcrypt registered [ 1019.547452] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:54:08 (1788310448) [ 1020.221452] Key type .llcrypt unregistered [ 1020.223232] Key type ._llcrypt unregistered [ 1026.945597] Key type ._llcrypt registered [ 1026.948378] Key type .llcrypt registered [ 1027.158484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.192757] alg: No test for adler32 (adler32-zlib) [ 1028.155340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1028.185937] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1035.335919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1045.320599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.392867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1045.443644] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1047.137321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_interrupt -r 1 [ 1047.225366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_interrupt -r 1 [ 1047.302566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_interrupt -r 1 [ 1047.350496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_interrupt -r 1 [ 1047.404689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_interrupt -r 1 [ 1047.454709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_interrupt -r 1 [ 1047.500890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_interrupt -r 1 [ 1047.557953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_interrupt -r 1 [ 1047.620799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1047.647654] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1047.652952] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1047.929790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1047.962773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1048.001982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.042839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.083792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.124942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.170857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.219830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.240569] LNet: 60609:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1048.265973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.301992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.340978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.374806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.420317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.467940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1048.507183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1048.982869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_dropped -r 1 [ 1049.009388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_dropped -r 1 [ 1049.039610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_dropped -r 1 [ 1049.081697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_dropped -r 1 [ 1049.117363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_dropped -r 1 [ 1049.159864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_dropped -r 1 [ 1049.188403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_dropped -r 1 [ 1049.226623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_dropped -r 1 [ 1049.254649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1049.272325] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1049.277403] LNet: Skipped 2 previous similar messages [ 1049.451668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1049.484125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1049.503945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1049.520654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1049.543574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1049.564465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1049.586467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1049.619823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1049.632813] LNet: 60730:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1049.653902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1049.688412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1049.728243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1049.753887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.775316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.801316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.835378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1050.411386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_aborted -r 1 [ 1050.451303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_aborted -r 1 [ 1050.507439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_aborted -r 1 [ 1050.542924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_aborted -r 1 [ 1050.578702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_aborted -r 1 [ 1050.623149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_aborted -r 1 [ 1050.688426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_aborted -r 1 [ 1050.729268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_aborted -r 1 [ 1050.766409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1050.787203] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1050.791826] LNet: Skipped 2 previous similar messages [ 1050.944957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1050.968195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1050.990427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.014584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.040882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.061598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.091619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.112855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.134751] LNet: 60851:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1051.154498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.181304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1051.211513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1051.236831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1051.272404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1051.318885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.359928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1051.805912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_no_route -r 1 [ 1051.845384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_no_route -r 1 [ 1051.875482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_no_route -r 1 [ 1051.911261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_no_route -r 1 [ 1051.934408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_no_route -r 1 [ 1051.956643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_no_route -r 1 [ 1051.982283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_no_route -r 1 [ 1052.019696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_no_route -r 1 [ 1052.046962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1052.283841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.308790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.332501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.358378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.382773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.405745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.428546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.448951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.465296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.478637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.495472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.508593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.522358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.541475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.556440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1053.008721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_timeout -r 1 [ 1053.035802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_timeout -r 1 [ 1053.068445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_timeout -r 1 [ 1053.134442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_timeout -r 1 [ 1053.207160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_timeout -r 1 [ 1053.298325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_timeout -r 1 [ 1053.343176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_timeout -r 1 [ 1053.384781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_timeout -r 1 [ 1053.428717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1053.452922] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1053.457598] LNet: Skipped 5 previous similar messages [ 1053.673530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.698574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1053.720986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.744806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.768384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.793494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.823343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.852344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.868426] LNet: 61094:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.873171] LNet: 61094:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1053.895253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.917086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.935855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.949835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.979853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.005749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.038362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1054.448341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_error -r 1 [ 1054.478466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_error -r 1 [ 1054.519483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_error -r 1 [ 1054.564885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_error -r 1 [ 1054.608953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_error -r 1 [ 1054.632433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_error -r 1 [ 1054.667534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_error -r 1 [ 1054.698864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_error -r 1 [ 1054.722685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1054.988832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.021226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1055.041470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.081722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.120828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.162638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.202951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.244516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.271511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1055.311624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1055.361260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1055.398071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1055.460963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1055.484042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1055.507790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1061.372393] LNet: 61295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.381697] LNetError: 61295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.392518] LNet: Removed LNI 192.168.201.42@tcp [ 1061.414993] LNet: Removed LNI 192.168.201.42@tcp1 [ 1061.963593] Key type .llcrypt unregistered [ 1061.965492] Key type ._llcrypt unregistered [ 1081.010370] Key type ._llcrypt registered [ 1081.012893] Key type .llcrypt registered [ 1082.632964] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:55:10 (1788310510) [ 1083.295325] Key type .llcrypt unregistered [ 1083.301664] Key type ._llcrypt unregistered [ 1090.278461] Key type ._llcrypt registered [ 1090.280488] Key type .llcrypt registered [ 1090.511600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.521476] alg: No test for adler32 (adler32-zlib) [ 1091.401717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1091.419585] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1099.749870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1100.538756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_dropped -r 1 [ 1100.572813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_dropped -r 1 [ 1100.613164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1100.645756] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1100.941808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.990973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.029178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.093358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.156700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.215817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.316309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.393728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.416958] LNet: 62549:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1101.439868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.486452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.543928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.594904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.639947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.681473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.739017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1102.333610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_error -r 1 [ 1102.364278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_error -r 1 [ 1102.418451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1102.450836] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1102.710994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.766190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1102.823430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.862338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.887380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.913390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.962799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.020492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.034460] LNet: 62658:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.053949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.080116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.107259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.144703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.190694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.275773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.321333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.799552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_timeout -r 1 [ 1103.827693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_timeout -r 1 [ 1103.886618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1103.908555] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1104.166516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.191796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.220455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.239260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.260386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.282766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.304819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.330529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.346280] LNet: 62767:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.361271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.384390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.407979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.430966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.455857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.505682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.559458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.944561] LNet: 62812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.960193] LNetError: 62812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.990594] LNet: Removed LNI 192.168.201.42@tcp [ 1105.493503] Key type .llcrypt unregistered [ 1105.500236] Key type ._llcrypt unregistered [ 1124.896163] Key type ._llcrypt registered [ 1124.903113] Key type .llcrypt registered [ 1126.322474] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:55:54 (1788310554) [ 1127.236303] Key type .llcrypt unregistered [ 1127.240279] Key type ._llcrypt unregistered [ 1135.489598] Key type ._llcrypt registered [ 1135.492792] Key type .llcrypt registered [ 1135.766617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1135.792993] alg: No test for adler32 (adler32-zlib) [ 1136.754942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1136.785694] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1144.623836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1155.183921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.290113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1155.351676] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1156.975388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_dropped -r 1 [ 1157.027371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e remote_dropped -r 1 [ 1157.086895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_dropped -r 1 [ 1157.123103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e remote_dropped -r 1 [ 1157.178504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e remote_dropped -r 1 [ 1157.228401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e remote_dropped -r 1 [ 1157.291936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e remote_dropped -r 1 [ 1157.336811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e remote_dropped -r 1 [ 1157.428433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1157.460208] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1157.479392] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.142@tcp [ 1157.792604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1157.865993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1157.947816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.029402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.096930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.179681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.231650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.296175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.349698] LNet: 64283:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.371918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.432467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.495382] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1158.502592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.507875] LNet: Skipped 3 previous similar messages [ 1158.568655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1158.622670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1158.664625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1158.735137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1158.769432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.333738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_error -r 1 [ 1159.399183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e remote_error -r 1 [ 1159.465351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_error -r 1 [ 1159.536301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e remote_error -r 1 [ 1159.602972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e remote_error -r 1 [ 1159.672643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e remote_error -r 1 [ 1159.746700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e remote_error -r 1 [ 1159.782477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e remote_error -r 1 [ 1159.813758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1159.830962] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1160.111704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.150547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.194495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.232424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.276718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.340984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.407616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.470077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.491359] LNet: 64406:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.518470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.554650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.633928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.682200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.725402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.795675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.872328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1160.942621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.857657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_timeout -r 1 [ 1161.937297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e remote_timeout -r 1 [ 1161.992809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_timeout -r 1 [ 1162.047175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e remote_timeout -r 1 [ 1162.090962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e remote_timeout -r 1 [ 1162.163658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e remote_timeout -r 1 [ 1162.216502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e remote_timeout -r 1 [ 1162.288537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e remote_timeout -r 1 [ 1162.353729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1162.386481] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1162.658952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.689513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1162.730952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.773476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.804868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.839304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.879584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.915830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.941575] LNet: 64530:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.970320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.008624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.041921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.070575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.102641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.144279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.212833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1163.277351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.782811] LNet: 64612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.793144] LNetError: 64612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.812865] LNet: Removed LNI 192.168.201.42@tcp [ 1168.833717] LNet: Removed LNI 192.168.201.42@tcp1 [ 1169.232792] Key type .llcrypt unregistered [ 1169.239279] Key type ._llcrypt unregistered [ 1189.483417] Key type ._llcrypt registered [ 1189.484882] Key type .llcrypt registered [ 1191.230568] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:56:59 (1788310619) [ 1193.358191] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1195.408774] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:57:03 (1788310623) [ 1196.648322] Key type .llcrypt unregistered [ 1196.650254] Key type ._llcrypt unregistered [ 1203.871738] Key type ._llcrypt registered [ 1203.876129] Key type .llcrypt registered [ 1204.203542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1204.226401] alg: No test for adler32 (adler32-zlib) [ 1205.229459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1205.271708] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1214.140961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1215.109740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e network_timeout -r 1 [ 1215.163436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e network_timeout -r 1 [ 1215.242800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1215.288048] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1215.528066] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1215.806439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1215.872793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1215.894893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.923123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.971830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.051093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.103767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.154770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.189694] LNet: 66161:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.213569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.264865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.297836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.366531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1216.406410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.444421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.808494] LNet: 66204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.827466] LNetError: 66204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.840602] LNet: Removed LNI 192.168.201.42@tcp [ 1217.373512] Key type .llcrypt unregistered [ 1217.380591] Key type ._llcrypt unregistered [ 1240.135862] Key type ._llcrypt registered [ 1240.138541] Key type .llcrypt registered [ 1240.387338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1240.432913] alg: No test for adler32 (adler32-zlib) [ 1241.395466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1241.417352] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1248.593211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1258.749823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1258.835776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1258.855887] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1260.159951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e network_timeout -r 1 [ 1260.187095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e network_timeout -r 1 [ 1260.212958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e network_timeout -r 1 [ 1260.241262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e network_timeout -r 1 [ 1260.279281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e network_timeout -r 1 [ 1260.320482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e network_timeout -r 1 [ 1260.379416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e network_timeout -r 1 [ 1260.412268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e network_timeout -r 1 [ 1260.446563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1260.463621] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1260.632322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.664178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1260.688462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.729600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.762581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1260.778908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1260.803702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1260.823888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1260.838362] LNet: 67088:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1260.856205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1260.881641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1260.903203] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1260.905652] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.142@tcp [ 1260.906854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1260.936744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1260.963243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.988153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1264.843066] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1265.058066] LNet: 67165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.061855] LNetError: 67165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.083526] LNet: Removed LNI 192.168.201.42@tcp [ 1265.103536] LNet: Removed LNI 192.168.201.42@tcp1 [ 1265.545973] Key type .llcrypt unregistered [ 1265.548223] Key type ._llcrypt unregistered [ 1284.639770] Key type ._llcrypt registered [ 1284.641351] Key type .llcrypt registered [ 1286.119826] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:58:34 (1788310714) [ 1286.758676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1286.769758] alg: No test for adler32 (adler32-zlib) [ 1287.715220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.797872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1287.844806] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1288.768355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1288.791606] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1289.711271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 1289.780838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1289.814891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1289.865426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1289.912308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1289.958681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1290.010618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.42@tcp [ 1290.048352] LNetError: 68266:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.42@tcp to recovery [ 1290.847299] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1290.850762] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1292.897365] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1293.608560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1296.998962] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1297.239438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1297.261629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1297.328436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1297.346546] LNet: 68373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.351625] LNetError: 68373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.399184] LNet: Removed LNI 192.168.201.42@tcp [ 1298.405806] LNetError: 68373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.427367] LNet: Removed LNI 192.168.201.42@tcp1 [ 1298.475669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1298.572166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1298.596117] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1299.574895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1299.587823] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1300.382702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 1300.411573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1300.485789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1300.557214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1300.611270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1300.647787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1300.686272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1300.720349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.42@tcp [ 1300.739392] LNetError: 68645:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.42@tcp to recovery [ 1301.602061] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1304.328666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1307.743370] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1307.748305] LNet: Skipped 1 previous similar message [ 1312.455113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1312.499318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1312.549986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1319.785534] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:59:08 (1788310748) [ 1320.355589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1320.374657] LNet: 69362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1320.380401] LNetError: 69362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1320.384916] LNet: Removed LNI 192.168.201.42@tcp [ 1320.399606] LNet: Removed LNI 192.168.201.42@tcp1 [ 1320.424228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1320.477534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1320.503679] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1321.447882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1321.466436] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1322.028696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 1322.106593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1322.144321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1322.195667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1322.236769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1322.269490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1322.301412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.42@tcp --health 0 [ 1322.404775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1322.527602] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.42@tcp [ 1323.567465] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1323.570576] LNet: Skipped 1 previous similar message [ 1338.071214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1338.232572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1338.262984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1338.302742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.42@tcp --health 500 [ 1338.426571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1345.158441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1345.217351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1345.243814] LNet: 69854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.262980] LNetError: 69854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.271766] LNetError: 69854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1345.288396] LNet: Removed LNI 192.168.201.42@tcp [ 1345.303736] LNet: Removed LNI 192.168.201.42@tcp1 [ 1345.326610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.390784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1345.416448] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1346.168560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1346.195662] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1347.055881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 1347.130733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1347.243153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1347.289619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1347.356259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1347.412469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1347.460761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1347.531303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.42@tcp --health 0 [ 1348.450075] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1348.453785] LNet: Skipped 3 previous similar messages [ 1348.982688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1359.329266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1359.365266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1359.406500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1365.215258] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:59:54 (1788310794) [ 1365.559137] LNet: 70899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.563129] LNetError: 70899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.566330] LNetError: 70899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1366.623448] LNet: Removed LNI 192.168.201.42@tcp [ 1366.646740] LNet: Removed LNI 192.168.201.42@tcp1 [ 1367.315521] Key type .llcrypt unregistered [ 1367.318153] Key type ._llcrypt unregistered [ 1372.668985] Key type ._llcrypt registered [ 1372.674953] Key type .llcrypt registered [ 1372.950862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1372.969148] alg: No test for adler32 (adler32-zlib) [ 1373.925647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1373.943992] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1381.683555] Lustre: DEBUG MARKER: Initial discovery [ 1381.744539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.142@tcp [ 1383.785827] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1383.992191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1384.012739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.142@tcp [ 1384.024297] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1384.039320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1384.063508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.142@tcp -r 1 -m GET -l 3 [ 1384.094989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.142@tcp -r 1 -m GET -e local_error [ 1384.115874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.142@tcp -r 1 -m PUT -l 6 [ 1384.224220] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1385.241378] Lustre: DEBUG MARKER: Force oleg142-client.virtnet to discover 192.168.201.142@tcp (in background) [ 1385.346555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.142@tcp [ 1386.430602] Lustre: DEBUG MARKER: Force oleg142-server to discover 192.168.201.42@tcp [ 1388.389051] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1388.826987] Lustre: DEBUG MARKER: Wait for 71664 [ 1390.268432] Lustre: DEBUG MARKER: Finished wait on 71664 [ 1390.367812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1390.402138] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1390.413203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1390.768379] LNet: 72119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.779337] LNetError: 72119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.805153] LNet: Removed LNI 192.168.201.42@tcp [ 1391.304686] Key type .llcrypt unregistered [ 1391.306198] Key type ._llcrypt unregistered [ 1411.981604] Key type ._llcrypt registered [ 1411.985492] Key type .llcrypt registered [ 1414.188127] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:00:41 (1788310841) [ 1415.343394] Key type .llcrypt unregistered [ 1415.350451] Key type ._llcrypt unregistered [ 1424.757111] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1425.005602] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1425.019113] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1425.021554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1427.778797] Key type ._llcrypt registered [ 1427.781748] Key type .llcrypt registered [ 1428.117183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1428.129443] alg: No test for adler32 (adler32-zlib) [ 1429.213363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.267786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1429.321059] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1430.439807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1430.461134] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1440.486653] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:01:08 (1788310868) [ 1440.589793] LNet: 74207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.602264] LNetError: 74207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1440.617194] LNet: Removed LNI 10.1.2.3@tcp [ 1440.661223] LNet: Removed LNI 192.168.201.42@tcp [ 1441.353632] Key type .llcrypt unregistered [ 1441.360899] Key type ._llcrypt unregistered [ 1446.219909] Key type ._llcrypt registered [ 1446.222421] Key type .llcrypt registered [ 1446.376858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1446.392848] alg: No test for adler32 (adler32-zlib) [ 1447.320225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1447.362472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1447.383056] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1447.963479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1447.985797] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1448.747611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.42@tcp [ 1449.823248] LNetError: Unexpected error -22 connecting to 192.168.201.42@tcp at host 192.168.201.42:988 [ 1454.990943] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:01:23 (1788310883) [ 1455.390759] LNet: 75351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1455.397333] LNetError: 75351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1455.404875] LNet: Removed LNI 10.1.2.3@tcp [ 1455.413767] LNet: Removed LNI 192.168.201.42@tcp [ 1456.076601] Key type .llcrypt unregistered [ 1456.078318] Key type ._llcrypt unregistered [ 1463.234529] Key type ._llcrypt registered [ 1463.236710] Key type .llcrypt registered [ 1463.634116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1463.666644] alg: No test for adler32 (adler32-zlib) [ 1464.696217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.760662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1464.807580] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1466.023680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1466.055082] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 1467.250971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.42@tcp1 --nid 192.168.201.42@tcp2 [ 1473.153756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp1 192.168.201.42@tcp2 [ 1473.200825] LNetError: 76192:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.42@tcp1 to 192.168.201.42@tcp2 - Route table may be misconfigured [ 1473.236063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp2 192.168.201.42@tcp1 [ 1481.706371] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:01:50 (1788310910) [ 1482.062081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.100630] LNet: 76765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.108467] LNetError: 76765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1482.114968] LNet: Removed LNI 192.168.201.42@tcp1 [ 1483.180931] LNet: Removed LNI 192.168.201.42@tcp2 [ 1483.221741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.254909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.265800] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1483.818413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1483.836699] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1484.717349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 1484.778412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.42@tcp [ 1484.878829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp1 192.168.201.42@tcp1 [ 1484.976812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.42@tcp -d 192.168.201.42@tcp -e network_timeout [ 1485.026828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -e network_timeout [ 1485.097810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -e network_timeout [ 1485.177093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -e network_timeout [ 1485.220744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1485.256580] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1485.270900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1485.279110] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1485.347400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1485.480849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1491.500876] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:02:00 (1788310920) [ 1491.732134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1491.747216] LNet: 77628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.751771] LNetError: 77628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.758822] LNetError: 77628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1491.764737] LNet: Removed LNI 192.168.201.42@tcp [ 1492.839734] LNet: Removed LNI 192.168.201.42@tcp1 [ 1492.874597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.997079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1493.317191] LNet: 77669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.727962] Key type .llcrypt unregistered [ 1493.731557] Key type ._llcrypt unregistered [ 1507.054737] Key type ._llcrypt registered [ 1507.060473] Key type .llcrypt registered [ 1508.524253] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:02:16 (1788310936) [ 1508.835650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1508.846794] alg: No test for adler32 (adler32-zlib) [ 1509.822511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.220904] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1513.277736] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:02:21 (1788310941) [ 1513.369601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.392315] LNet: 78760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.420273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.502544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.551614] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1514.517885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1514.532913] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1515.140175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1515.168418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp1 [ 1515.206986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp1 [ 1521.903883] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:02:29 (1788310949) [ 1523.344981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg142-server [ 1525.466362] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:02:33 (1788310953) [ 1527.581960] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg142-server [ 1529.729992] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:02:37 (1788310957) [ 1532.043613] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg142-server [ 1533.945253] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:02:42 (1788310962) [ 1536.016394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg142-server [ 1538.308553] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:02:46 (1788310966) [ 1540.126155] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg142-server [ 1541.972808] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:02:50 (1788310970) [ 1543.838354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg142-server [ 1545.462764] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:02:54 (1788310974) [ 1547.604247] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg142-server [ 1549.298720] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:02:57 (1788310977) [ 1551.232372] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg142-server [ 1552.792974] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:03:01 (1788310981) [ 1554.941712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg142-server [ 1556.727897] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:03:05 (1788310985) [ 1556.985816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.021939] LNet: 82646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.041065] LNetError: 82646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.050222] LNet: Removed LNI 192.168.201.42@tcp [ 1558.118606] LNet: Removed LNI 192.168.201.42@tcp1 [ 1558.182561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.282835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1558.327756] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1559.229973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1559.287667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1559.449154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.471670] LNet: 82793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.481748] LNetError: 82793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.486537] LNetError: 82793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1559.499688] LNet: Removed LNI 192.168.201.42@tcp [ 1559.516572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.605844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1559.667437] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1560.487734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1560.615693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.648220] LNetError: 82938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.667894] LNet: Removed LNI 192.168.201.42@tcp [ 1560.685345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.725322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1560.745786] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1561.294928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1561.359510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1561.574419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.593327] LNet: 83085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.595568] LNet: 83085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1562.663478] LNet: Removed LNI 192.168.201.42@tcp [ 1562.709618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.796338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1562.829854] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1563.635444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1563.953368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.966837] LNetError: 83231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.971927] LNetError: 83231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1563.984914] LNet: Removed LNI 192.168.201.42@tcp [ 1564.002921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.047955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1564.070994] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1564.793664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1564.858602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1564.974676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.060951] LNet: Removed LNI 192.168.201.42@tcp [ 1566.089392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.176961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1566.207361] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1567.084182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1567.320081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.355973] LNet: 83523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.361734] LNet: 83523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1568.424425] LNet: Removed LNI 192.168.201.42@tcp [ 1568.484502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.537573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1568.555933] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1569.395502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1569.555717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1569.831335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.870009] LNetError: 83671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.882034] LNetError: 83671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1569.911452] LNet: Removed LNI 192.168.201.42@tcp [ 1569.942956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.034254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1570.069417] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1570.847767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1571.056582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.142567] LNet: Removed LNI 192.168.201.42@tcp [ 1572.187497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.236319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1572.266224] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1572.989792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1573.043107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1573.289152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.380640] LNet: Removed LNI 192.168.201.42@tcp [ 1574.423545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.504449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1574.534266] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1575.536128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1578.060229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.075920] LNet: 84127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.079399] LNet: 84127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1578.084607] LNetError: 84127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1578.087046] LNetError: 84127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1578.099820] LNet: Removed LNI 192.168.201.42@tcp [ 1578.116931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.165334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.188529] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1578.940612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1578.985876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1579.205831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.265484] LNet: Removed LNI 192.168.201.42@tcp [ 1580.343951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.416487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1580.439860] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1581.395608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1581.574417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.633872] LNet: Removed LNI 192.168.201.42@tcp [ 1582.667719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.743646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.789278] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1583.547641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1583.603400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.649191] LNet: Removed LNI 192.168.201.42@tcp [ 1583.668435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.726374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1583.748118] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1584.630897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1584.683343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.708404] LNet: Removed LNI 192.168.201.42@tcp [ 1584.723494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.765520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1584.839109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.867461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.933981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1585.006444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.037511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.118233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1585.239361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.280514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.337739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1585.356310] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1586.119530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.171095] LNet: Removed LNI 192.168.201.42@tcp [ 1586.198786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.302885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1592.819230] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:03:41 (1788311021) [ 1593.051396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.119584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.186553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1593.210901] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/180] [ 1594.318221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.338759] LNet: 85578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.344284] LNet: 85578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1594.351333] LNetError: 85578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.356138] LNetError: 85578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1594.366659] LNet: Removed LNI 192.168.201.42@tcp231 [ 1594.379387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.426674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1594.452085] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/180] [ 1594.456371] LNet: 85586:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1594.933583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1594.977851] LNet: Removed LNI 192.168.201.42@tcp231 [ 1594.995513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1595.039022] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/180] [ 1596.146626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.168255] LNet: Removed LNI 192.168.201.42@tcp231 [ 1596.187618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.241619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1596.255356] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/0] [ 1596.259256] LNet: 85739:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1596.684268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1596.726511] LNet: Removed LNI 192.168.201.42@tcp231 [ 1596.757329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1596.794448] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/0] [ 1597.961440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.991861] LNet: Removed LNI 192.168.201.42@tcp231 [ 1598.018726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.063877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1598.102453] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/60] [ 1598.110995] LNet: 85890:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1598.685975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1598.713897] LNet: Removed LNI 192.168.201.42@tcp231 [ 1598.724926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1598.747298] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/60] [ 1599.764217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.812560] LNet: Removed LNI 192.168.201.42@tcp231 [ 1599.832577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.923100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1599.963398] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/180] [ 1600.420334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1600.472605] LNet: Removed LNI 192.168.201.42@tcp231 [ 1600.497473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1600.530635] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/180] [ 1606.802200] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:03:55 (1788311035) [ 1606.959938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.023965] LNet: Removed LNI 192.168.201.42@tcp231 [ 1607.086818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.149309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1607.167585] LNet: Added LNI 192.168.201.42@tcp232 [8/256/0/180] [ 1607.810807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1607.848929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1607.876533] LNet: Removed LNI 192.168.201.42@tcp232 [ 1607.888549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1607.916083] LNet: Added LNI 192.168.201.42@tcp232 [8/256/0/180] [ 1608.754826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1608.820653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1608.847219] LNet: Removed LNI 192.168.201.42@tcp232 [ 1608.858973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1608.876619] LNet: Added LNI 192.168.201.42@tcp232 [8/256/0/180] [ 1609.517725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1609.552630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1609.590313] LNet: Removed LNI 192.168.201.42@tcp232 [ 1609.614642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1609.665531] LNet: Added LNI 192.168.201.42@tcp232 [8/256/0/180] [ 1610.695322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1610.769639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1610.801963] LNet: Removed LNI 192.168.201.42@tcp232 [ 1610.819937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1616.987287] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:04:05 (1788311045) [ 1618.350280] Key type .llcrypt unregistered [ 1618.351891] Key type ._llcrypt unregistered [ 1624.718441] Key type ._llcrypt registered [ 1624.730440] Key type .llcrypt registered [ 1625.001696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1625.009139] alg: No test for adler32 (adler32-zlib) [ 1625.930491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1625.986337] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1633.450733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1643.400270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.427707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1643.440035] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1645.528089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_interrupt -r 2 [ 1645.559139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_interrupt -r 2 [ 1645.649324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_interrupt -r 2 [ 1645.692676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_interrupt -r 2 [ 1645.736445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_interrupt -r 2 [ 1645.788680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_interrupt -r 2 [ 1645.898975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_interrupt -r 2 [ 1645.960774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_interrupt -r 2 [ 1646.001285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1646.017294] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1646.022345] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1646.267945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1646.298756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1646.323538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1646.351497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1646.372233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1646.387796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1646.414620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1646.440722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1646.454503] LNet: 88545:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1646.473662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1646.496936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1646.523439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1646.601774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1646.657243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1646.684134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1647.217982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_dropped -r 2 [ 1647.284878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_dropped -r 2 [ 1647.323512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_dropped -r 2 [ 1647.386965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_dropped -r 2 [ 1647.445425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_dropped -r 2 [ 1647.497468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_dropped -r 2 [ 1647.538982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_dropped -r 2 [ 1647.587190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_dropped -r 2 [ 1647.632238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1647.672284] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1648.010995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1648.085799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1648.149082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1648.205457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1648.256307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1648.288227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1648.324931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1648.373557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1648.393371] LNet: 88665:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1648.425427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1648.470965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1648.525404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1648.573172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1648.614825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1648.665603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1649.413387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_aborted -r 2 [ 1649.451046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_aborted -r 2 [ 1649.485935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_aborted -r 2 [ 1649.523699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_aborted -r 2 [ 1649.551500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_aborted -r 2 [ 1649.578443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_aborted -r 2 [ 1649.613562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_aborted -r 2 [ 1649.685966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_aborted -r 2 [ 1649.755622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1649.782941] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1650.164633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1650.226676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1650.270505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1650.312714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1650.352386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1650.403655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1650.445989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1650.484436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1650.514469] LNet: 88785:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1650.564326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1650.603472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1650.637510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1650.664067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1650.695803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1650.741962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1651.377242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_no_route -r 2 [ 1651.411922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_no_route -r 2 [ 1651.467451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_no_route -r 2 [ 1651.511547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_no_route -r 2 [ 1651.551877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_no_route -r 2 [ 1651.590465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_no_route -r 2 [ 1651.628604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_no_route -r 2 [ 1651.678557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_no_route -r 2 [ 1651.748389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1651.788432] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1652.014643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1652.082296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1652.136850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1652.225308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1652.304440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1652.332396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1652.401789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1652.461519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1652.503471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1652.542421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1652.575110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1652.596557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1652.640531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1652.702962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1653.379880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_timeout -r 2 [ 1653.401485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_timeout -r 2 [ 1653.418311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_timeout -r 2 [ 1653.443801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_timeout -r 2 [ 1653.477113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_timeout -r 2 [ 1653.515259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_timeout -r 2 [ 1653.541822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_timeout -r 2 [ 1653.569892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_timeout -r 2 [ 1653.604979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1653.913315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1653.978944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1654.035831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1654.066512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1654.100385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1654.137805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1654.163630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1654.204530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1654.217435] LNet: 89026:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1654.220471] LNet: 89026:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1654.238406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1654.260538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1654.297566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1654.325293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1654.349851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1654.382691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1654.960807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_dropped -r 2 [ 1655.014842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e remote_dropped -r 2 [ 1655.076758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_dropped -r 2 [ 1655.126049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e remote_dropped -r 2 [ 1655.181804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e remote_dropped -r 2 [ 1655.235277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e remote_dropped -r 2 [ 1655.325764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e remote_dropped -r 2 [ 1655.384409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e remote_dropped -r 2 [ 1655.452993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1655.500407] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.142@tcp [ 1655.812552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1655.858269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1655.896766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1655.961238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1655.995598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1656.034900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1656.062731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1656.123382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1656.181303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1656.275924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1656.310424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1656.335995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1656.373984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1656.448031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1662.236400] LNet: 89224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.243619] LNetError: 89224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.250425] LNet: Removed LNI 192.168.201.42@tcp [ 1662.265497] LNet: Removed LNI 192.168.201.42@tcp1 [ 1662.611682] Key type .llcrypt unregistered [ 1662.621456] Key type ._llcrypt unregistered [ 1677.835301] Key type ._llcrypt registered [ 1677.837367] Key type .llcrypt registered [ 1678.896958] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:05:07 (1788311107) [ 1679.233770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1679.255153] alg: No test for adler32 (adler32-zlib) [ 1680.161814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.211300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1680.233642] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1681.122947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1681.146790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.42@tcp [ 1681.167569] LNetError: 90202:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.42@tcp to recovery [ 1681.249360] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1681.297974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.42@tcp [ 1681.393545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1687.145878] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:05:15 (1788311115) [ 1687.263140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.290803] LNet: 90790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.305551] LNetError: 90790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.346517] LNet: Removed LNI 192.168.201.42@tcp [ 1687.365981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.418899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.436812] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1688.371578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1688.396174] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1689.335246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.42@tcp [ 1689.349483] LNetError: 91058:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.42@tcp to recovery [ 1689.443686] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1690.533981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.42@tcp [ 1697.680461] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:05:26 (1788311126) [ 1699.478599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg142-server [ 1701.011632] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:05:29 (1788311129) [ 1702.832232] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg142-server [ 1704.541522] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:05:32 (1788311132) [ 1706.450178] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg142-server [ 1707.889942] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:05:36 (1788311136) [ 1708.043055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.082980] LNet: 92678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.107191] LNetError: 92678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.119154] LNet: Removed LNI 192.168.201.42@tcp [ 1708.131614] LNet: Removed LNI 192.168.201.42@tcp1 [ 1708.148691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.222307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.239563] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1708.827061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1708.890962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1708.918879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1708.990485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1709.058163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1709.103639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1715.066346] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:05:43 (1788311143) [ 1715.211578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.258705] LNet: 93399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.283952] LNetError: 93399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.297073] LNetError: 93399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1715.334104] LNet: Removed LNI 192.168.201.42@tcp [ 1715.358315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.442105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.466095] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1723.306629] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:05:51 (1788311151) [ 1723.443627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.478928] LNet: 94103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.485544] LNetError: 94103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.497851] LNet: Removed LNI 192.168.201.42@tcp [ 1723.516276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.578759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1723.597146] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1730.806203] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:05:59 (1788311159) [ 1732.593677] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1734.173797] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:06:02 (1788311162) [ 1734.640888] LNet: 95145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.645967] LNetError: 95145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.661130] LNet: Removed LNI 192.168.201.42@tcp [ 1735.281754] Key type .llcrypt unregistered [ 1735.289528] Key type ._llcrypt unregistered [ 1741.294783] Key type ._llcrypt registered [ 1741.297339] Key type .llcrypt registered [ 1741.455447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1741.469722] alg: No test for adler32 (adler32-zlib) [ 1742.320175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1742.339473] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1748.804389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1751.794182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.142@tcp [ 1752.045665] LNet: 95570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.050513] LNetError: 95570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.063700] LNet: Removed LNI 192.168.201.42@tcp [ 1752.299209] Key type .llcrypt unregistered [ 1752.300310] Key type ._llcrypt unregistered [ 1763.928506] Key type ._llcrypt registered [ 1763.932147] Key type .llcrypt registered [ 1765.282936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1767.075578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1768.540312] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:06:37 (1788311197) [ 1770.269648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1772.059367] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:06:40 (1788311200) [ 1774.026288] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg142-server [ 1775.449507] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:06:44 (1788311204) [ 1777.063464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg142-server [ 1778.321177] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:06:47 (1788311207) [ 1779.568660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1779.592271] alg: No test for adler32 (adler32-zlib) [ 1780.689887] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1780.815701] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1782.447165] Key type lgssc registered [ 1783.496788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1806.410512] Key type lgssc unregistered [ 1806.581465] LNet: 98750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.585662] LNetError: 98750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.604517] LNet: Removed LNI 192.168.201.42@tcp [ 1807.001584] Key type .llcrypt unregistered [ 1807.002742] Key type ._llcrypt unregistered [ 1821.276427] Key type ._llcrypt registered [ 1821.278429] Key type .llcrypt registered [ 1822.284636] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:07:31 (1788311251) [ 1822.658576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1822.672921] alg: No test for adler32 (adler32-zlib) [ 1823.557878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.607610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1823.655525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1823.705117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1823.770249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1829.529603] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:07:38 (1788311258) [ 1829.685745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.721551] LNet: 100166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.755626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.812087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1829.843663] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1830.604490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.618841] LNet: 100311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.627912] LNetError: 100311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.639859] LNet: Removed LNI 192.168.201.42@tcp [ 1830.655801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.705785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1830.724273] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1831.201867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.223314] LNetError: 100339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.238217] LNet: Removed LNI 192.168.201.42@tcp [ 1831.255157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.295870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1831.319465] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1837.104757] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:07:45 (1788311265) [ 1837.360619] LNet: 100929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.363374] LNet: 100929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1837.368208] LNetError: 100929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.379896] LNet: Removed LNI 192.168.201.42@tcp [ 1838.017667] Key type .llcrypt unregistered [ 1838.021329] Key type ._llcrypt unregistered [ 1843.955948] Key type ._llcrypt registered [ 1843.957898] Key type .llcrypt registered [ 1844.213176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1844.235950] alg: No test for adler32 (adler32-zlib) [ 1845.109875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.206224] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1845.209164] LNet: *** cfs_fail_loc=e003, val=100*** [ 1846.027535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1846.044974] LNet: *** cfs_fail_loc=e003, val=100*** [ 1846.047778] LNet: Skipped 2 previous similar messages [ 1846.053685] Ping reply received with mapping: [ 1846.055575] found metadata reply NI[192.168.201.42@tcp] [ 1846.057672] 2a c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1846.064902] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1846.068207] 5a 5a 5a 5a 5a 5a 5a 5a [ 1846.206845] LNet: 101330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.213983] LNetError: 101330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.270781] LNet: Removed LNI 192.168.201.42@tcp [ 1847.859645] Key type .llcrypt unregistered [ 1847.861501] Key type ._llcrypt unregistered [ 1858.681782] Key type ._llcrypt registered [ 1858.685412] Key type .llcrypt registered [ 1859.773557] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:08:08 (1788311288) [ 1860.307586] Key type .llcrypt unregistered [ 1860.314455] Key type ._llcrypt unregistered [ 1866.615699] Key type ._llcrypt registered [ 1866.617443] Key type .llcrypt registered [ 1866.742910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1866.753498] alg: No test for adler32 (adler32-zlib) [ 1867.924036] LNetError: 102389:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1868.218550] Key type .llcrypt unregistered [ 1868.220291] Key type ._llcrypt unregistered [ 1872.993611] Key type ._llcrypt registered [ 1872.996249] Key type .llcrypt registered [ 1874.344141] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:08:22 (1788311302) [ 1875.108410] Key type .llcrypt unregistered [ 1875.110442] Key type ._llcrypt unregistered [ 1882.252916] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1882.266961] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1884.829861] Key type ._llcrypt registered [ 1884.832266] Key type .llcrypt registered [ 1884.938927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1884.954734] alg: No test for adler32 (adler32-zlib) [ 1886.032802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.133463] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1894.973542] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:08:43 (1788311323) [ 1895.131494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.184360] LNet: 104174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.192253] LNetError: 104174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1895.224432] LNet: Removed LNI 10.1.2.3@tcp [ 1895.241499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.927784] Key type .llcrypt unregistered [ 1895.930348] Key type ._llcrypt unregistered [ 1902.387452] Key type ._llcrypt registered [ 1902.390031] Key type .llcrypt registered [ 1902.561764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1902.577526] alg: No test for adler32 (adler32-zlib) [ 1903.562546] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1903.630388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1903.779453] LNet: 104469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.784578] LNetError: 104469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.814090] LNet: Removed LNI 192.168.201.42@tcp [ 1904.406457] Key type .llcrypt unregistered [ 1904.412864] Key type ._llcrypt unregistered [ 1915.278740] Key type ._llcrypt registered [ 1915.280494] Key type .llcrypt registered [ 1916.394256] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:09:05 (1788311345) [ 1916.867402] Key type .llcrypt unregistered [ 1916.868819] Key type ._llcrypt unregistered [ 1921.742569] Key type ._llcrypt registered [ 1921.744611] Key type .llcrypt registered [ 1921.877458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1921.889287] alg: No test for adler32 (adler32-zlib) [ 1927.302702] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:09:16 (1788311356) [ 1927.400493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1927.430488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.502123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1927.517896] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1928.218543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1928.253573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1928.288912] LNet: Removed LNI 192.168.201.42@tcp [ 1928.516877] LNet: 106224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.916135] Key type .llcrypt unregistered [ 1928.918706] Key type ._llcrypt unregistered [ 1939.398022] Key type ._llcrypt registered [ 1939.401560] Key type .llcrypt registered [ 1940.779267] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:09:29 (1788311369) [ 1941.141801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1941.155315] alg: No test for adler32 (adler32-zlib) [ 1942.036220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.088181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1942.118055] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1942.714044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1942.752542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.42@tcp [ 1942.762872] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1942.764395] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1942.766460] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1942, 0, 0 [ 1942.771152] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1942.772473] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1942.776138] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1942.779453] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1942, 0, 0 [ 1942.783496] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1942.786481] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1942.789195] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1942.791524] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1942, 0, 1992 [ 1942.795550] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1942.798062] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1942.801865] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1942.805847] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1942, 0, 1992 [ 1942.810122] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1948.171237] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:09:36 (1788311376) [ 1948.551934] LNet: 107763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.556460] LNetError: 107763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.572210] LNet: Removed LNI 192.168.201.42@tcp [ 1949.077433] Key type .llcrypt unregistered [ 1949.079834] Key type ._llcrypt unregistered [ 1954.564060] Key type ._llcrypt registered [ 1954.567948] Key type .llcrypt registered [ 1954.800874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1954.813137] alg: No test for adler32 (adler32-zlib) [ 1955.806805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1955.832465] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1964.965782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1975.782455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.798392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1975.811531] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1976.715211] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1976.787716] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1976.796515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1976.809592] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1979.231274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1979.257962] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1989.888633] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:10:18 (1788311418) [ 1990.073668] LNet: 109267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.100886] LNetError: 109267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.117510] LNet: Removed LNI 192.168.201.42@tcp [ 1991.200255] LNet: Removed LNI 192.168.201.42@tcp1 [ 1991.202815] LNetError: 109267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1991.206508] LNetError: 109267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1991.222366] LNet: Removed LNI 10.1.2.3@tcp99 [ 1991.992680] Key type .llcrypt unregistered [ 1991.994314] Key type ._llcrypt unregistered [ 1999.040118] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1999.060391] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2001.727745] Key type ._llcrypt registered [ 2001.732893] Key type .llcrypt registered [ 2001.924292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2001.936303] alg: No test for adler32 (adler32-zlib) [ 2002.847304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.910564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2002.934380] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2003.598713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2003.649567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2006.004783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2006.057488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2006.105749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2006.127271] LNetError: 110025:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2006.154834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2006.196794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2006.598294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2006.669982] LNet: 110045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.675886] LNetError: 110045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2006.684501] LNet: Removed LNI 10.1.2.3@tcp [ 2007.732754] LNet: Removed LNI 192.168.201.42@tcp [ 2008.408604] Key type .llcrypt unregistered [ 2008.410803] Key type ._llcrypt unregistered [ 2021.584169] Key type ._llcrypt registered [ 2021.585784] Key type .llcrypt registered [ 2022.940142] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:10:51 (1788311451) [ 2023.372535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.389498] alg: No test for adler32 (adler32-zlib) [ 2024.390857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.452284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2024.493327] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2025.133662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2025.207240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg142-client@tcp [ 2031.242942] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:10:59 (1788311459) [ 2031.327756] LNet: 111572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.332070] LNetError: 111572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.346110] LNet: Removed LNI 192.168.201.42@tcp [ 2032.039761] Key type .llcrypt unregistered [ 2032.042154] Key type ._llcrypt unregistered [ 2039.017569] Key type ._llcrypt registered [ 2039.023512] Key type .llcrypt registered [ 2039.219469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2039.246544] alg: No test for adler32 (adler32-zlib) [ 2040.172956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2040.198217] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2040.239488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2046.587232] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:11:15 (1788311475) [ 2048.126476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2049.485849] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:11:18 (1788311478) [ 2050.582724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2052.420460] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:11:20 (1788311480) [ 2052.512359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2052.540721] LNet: 113007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.550909] LNetError: 113007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.572194] LNet: Removed LNI 192.168.201.42@tcp [ 2052.590747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.703091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2052.738989] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2053.271968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2053.311713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2060.111313] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:11:28 (1788311488) [ 2060.246375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2060.269121] LNet: 113718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.279776] LNetError: 113718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.321095] LNet: Removed LNI 192.168.201.42@tcp [ 2060.378418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2060.468712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2060.502036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2061.142604] Key type .llcrypt unregistered [ 2061.144844] Key type ._llcrypt unregistered [ 2072.404605] Key type ._llcrypt registered [ 2072.407396] Key type .llcrypt registered [ 2073.736327] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:11:42 (1788311502) [ 2074.190639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2074.213621] alg: No test for adler32 (adler32-zlib) [ 2075.162659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2075.217931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2075.252676] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2075.908176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2075.963471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2082.017514] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:11:50 (1788311510) [ 2082.142966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2082.209365] LNet: 115278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.228049] LNetError: 115278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.256759] LNet: Removed LNI 192.168.201.42@tcp [ 2082.315556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.371887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2082.406198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2087.992439] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:11:56 (1788311516) [ 2088.220887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.243587] LNet: 115863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.258129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.308154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2088.616662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2088.654353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2088.703641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2089.131759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2089.155051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2089.218298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2089.244361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.284961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2089.637868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2089.653572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2089.692117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2090.040551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2090.078480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2095.746299] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:12:04 (1788311524) [ 2095.938531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2095.976312] LNet: 116497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2095.980624] LNet: 116497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2096.013383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.089284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.119607] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2096.830487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2096.848445] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2097.508440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2097.543542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2097.700941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2097.721365] LNetError: 116776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.783600] LNet: Removed LNI 192.168.201.42@tcp [ 2098.786351] LNetError: 116776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.817949] LNet: Removed LNI 192.168.201.42@tcp2 [ 2098.846524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.887523] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2098.895871] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2098.903659] LNet: 116783:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2099.230489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2099.373734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2099.391657] LNet: 116820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.395848] LNet: 116820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2100.447166] LNet: Removed LNI 192.168.201.42@tcp [ 2100.462261] LNet: Removed LNI 192.168.201.42@tcp2 [ 2100.480735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.527713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2100.545455] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2101.101947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2101.121901] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2101.758416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2101.789510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2101.822738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2102.031693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2102.051489] LNetError: 117104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.055606] LNetError: 117104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2102.059885] LNet: Removed LNI 192.168.201.42@tcp [ 2103.095173] LNet: Removed LNI 192.168.201.42@tcp2 [ 2103.119074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.209536] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2103.216879] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2103.226966] LNet: 117110:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2103.656588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2103.864841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2103.878785] LNet: 117147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.882730] LNet: 117147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2103.887754] LNet: Removed LNI 192.168.201.42@tcp [ 2104.938173] LNet: Removed LNI 192.168.201.42@tcp2 [ 2104.970777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.031683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2105.052317] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2105.724634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2105.736083] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2106.382517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2106.435911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2106.618876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2106.633489] LNetError: 117427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.643226] LNetError: 117427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2106.648723] LNet: Removed LNI 192.168.201.42@tcp [ 2106.658466] LNet: Removed LNI 192.168.201.42@tcp2 [ 2106.686584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.730281] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2106.734571] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2106.739384] LNet: 117433:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2107.369505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2107.588510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2108.640253] LNet: Removed LNI 192.168.201.42@tcp [ 2108.658903] LNet: Removed LNI 192.168.201.42@tcp2 [ 2108.704302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.802503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2108.817551] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2109.648067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2109.668436] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2110.405324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2110.436653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2110.504273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2110.707797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.775166] LNet: Removed LNI 192.168.201.42@tcp [ 2112.807647] LNet: Removed LNI 192.168.201.42@tcp2 [ 2112.844733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.944845] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2112.954227] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2112.958947] LNet: 117760:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2113.335367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2113.467437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.484611] LNet: 117798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.489534] LNet: 117798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2113.495948] LNet: Removed LNI 192.168.201.42@tcp [ 2114.535670] LNet: Removed LNI 192.168.201.42@tcp2 [ 2114.578789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.664724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2114.683069] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2115.119392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2115.138337] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2115.768661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2115.787635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2115.828434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2116.118219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.134548] LNetError: 118094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.137464] LNetError: 118094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2116.141352] LNet: Removed LNI 192.168.201.42@tcp [ 2116.149838] LNet: Removed LNI 192.168.201.42@tcp2 [ 2116.165335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.232448] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2116.242303] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2116.830363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2117.230375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.303218] LNet: Removed LNI 192.168.201.42@tcp [ 2118.325405] LNet: Removed LNI 192.168.201.42@tcp2 [ 2118.374359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.458672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2118.502497] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2119.211476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2119.238979] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2120.161920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2120.197868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2120.210957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2120.237547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2120.508733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.567158] LNet: Removed LNI 192.168.201.42@tcp [ 2122.598847] LNet: Removed LNI 192.168.201.42@tcp2 [ 2122.620757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.665032] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2122.670780] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2122.677903] LNet: 118460:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2122.683705] LNet: 118460:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2123.103129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2130.130199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2131.517348] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:12:40 (1788311560) [ 2131.616761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.665696] LNet: 119212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.673495] LNet: 119212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2131.682602] LNet: Removed LNI 192.168.201.42@tcp [ 2132.712264] LNet: Removed LNI 192.168.201.42@tcp2 [ 2132.739585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.000753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2133.022989] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2133.547655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2133.567360] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2134.341781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.359564] LNetError: 119493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2134.363337] LNetError: 119493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2134.367515] LNet: Removed LNI 10.1.2.3@tcp [ 2134.375884] LNet: Removed LNI 192.168.201.42@tcp [ 2134.389830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.445341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2134.467279] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2134.471556] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2134.933375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2134.958798] LNet: Removed LNI 192.168.201.42@tcp [ 2134.972925] LNet: Removed LNI 10.1.2.3@tcp [ 2139.920829] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 21:12:48 (1788311568) [ 2140.166698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.214977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.265916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.279165] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2140.922623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2140.961728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2140.990978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2141.021106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2141.081119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2141.150988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2141.187617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2141.223481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2141.256474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2141.287719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2141.318504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2141.461349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2146.983365] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:12:55 (1788311575) [ 2147.090570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.137991] LNet: Removed LNI 192.168.201.42@tcp [ 2148.161628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.256250] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2148.424868] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2148.428726] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2148.431298] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2150.703945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.727827] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2151.232615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2151.243824] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2152.993913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2153.027196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2153.084587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.42@tcp -m PUT -e local_timeout -r 1 [ 2153.147501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2153.230763] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2153.236370] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2153.411641] LNetError: 120901:0:(lib-move.c:4801:lnet_parse()) 192.168.201.42@tcp, src 192.168.201.42@tcp: Dropping PUT (error -108 looking up sender) [ 2154.465689] LNet: Removed LNI 10.1.2.3@tcp [ 2155.494195] LNet: Removed LNI 192.168.201.42@tcp [ 2155.939140] Key type .llcrypt unregistered [ 2155.942665] Key type ._llcrypt unregistered [ 2168.571971] Key type ._llcrypt registered [ 2168.576294] Key type .llcrypt registered [ 2170.096277] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:13:18 (1788311598) [ 2170.638914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2170.681713] alg: No test for adler32 (adler32-zlib) [ 2171.653829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.079925] LNet: 122216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.944114] Key type .llcrypt unregistered [ 2172.945838] Key type ._llcrypt unregistered [ 2179.823301] Key type ._llcrypt registered [ 2179.826334] Key type .llcrypt registered [ 2180.096599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2180.119711] alg: No test for adler32 (adler32-zlib) [ 2181.045119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2181.059217] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2188.763615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 2188.990856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2189.736152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.142@tcp [ 2189.993424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2190.019386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2190.045347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2190.093671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2190.113302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2190.128300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2190.150986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2190.185817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2190.217384] LNet: 122669:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2190.246495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2190.282915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2190.303274] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.142@tcp [ 2190.338340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2190.390410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2190.427920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2190.468384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2191.188929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2191.398296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2191.425402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2191.455933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2191.485368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2191.513919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2191.540861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2191.578427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2191.608592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2191.627157] LNet: 122787:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2191.650066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2191.702545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2191.726067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2191.751184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2191.790254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2191.821678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2192.519995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.142@tcp [ 2192.763959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2192.784843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2192.812570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2192.838676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2192.862972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2192.885644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2192.907405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2192.928630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2192.940630] LNet: 122904:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2192.956809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2192.987526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2193.022915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2193.055766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2193.100574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2193.151309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2193.814808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2194.141876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2194.179813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2194.229947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2194.267884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2194.313120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2194.342060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2194.392802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2194.423138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2194.455971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2194.494806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2194.517756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2194.548426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2194.579220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2194.607646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2194.915496] LNet: 123071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.940211] LNetError: 123071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2196.007338] LNet: Removed LNI 192.168.201.42@tcp [ 2196.418450] Key type .llcrypt unregistered [ 2196.423950] Key type ._llcrypt unregistered [ 2215.254374] Key type ._llcrypt registered [ 2215.255721] Key type .llcrypt registered [ 2216.376749] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:14:05 (1788311645) [ 2216.818784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2216.834968] alg: No test for adler32 (adler32-zlib) [ 2217.793796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.205566] LNet: 123960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.773473] Key type .llcrypt unregistered [ 2218.775688] Key type ._llcrypt unregistered [ 2224.762346] Key type ._llcrypt registered [ 2224.764965] Key type .llcrypt registered [ 2224.955989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2224.993353] alg: No test for adler32 (adler32-zlib) [ 2225.964099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2225.995886] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2232.655490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 2240.031774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.099542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2240.141405] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 2241.461960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2242.266335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.142@tcp [ 2242.411309] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.142@tcp [ 2242.545873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2242.604491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2242.683910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2242.720295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2242.740227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2242.767227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2242.827403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2242.898927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2242.939217] LNet: 124615:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2242.967496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2243.016144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2243.072226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2243.128329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2243.172328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2243.224669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2243.834178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2244.229592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2244.276393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2244.329643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2244.390277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2244.464878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2244.538441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2244.611238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2244.652241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2244.675438] LNet: 124732:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2244.688631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2244.734732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2244.768899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2244.802440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2244.851211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2244.904126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2245.721941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.142@tcp [ 2245.862391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2245.884295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2245.915940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2245.950891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2245.999860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2246.060934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2246.115625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2246.154521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2246.172653] LNet: 124850:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2246.189243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2246.247435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2246.267992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2246.299628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2246.349852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2246.414975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2247.633421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2248.041207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2248.111619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2248.171994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2248.203206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2248.246220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2248.275117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2248.302774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2248.335304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2248.348387] LNet: 124967:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2248.362334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2248.382441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2248.409723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2248.449303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2248.488946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2248.517079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2253.645454] LNet: 125052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.649975] LNetError: 125052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.659273] LNet: Removed LNI 192.168.201.42@tcp [ 2253.669495] LNet: Removed LNI 192.168.201.42@tcp1 [ 2254.106990] Key type .llcrypt unregistered [ 2254.111973] Key type ._llcrypt unregistered [ 2272.150472] Key type ._llcrypt registered [ 2272.153531] Key type .llcrypt registered [ 2273.757950] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 21:15:01 (1788311701) [ 2274.118778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.135670] alg: No test for adler32 (adler32-zlib) [ 2275.053492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.098260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2275.117887] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2275.716306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2275.744980] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2276.249754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2276.298753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2277.344298] LNet: Removed LNI 192.168.201.42@tcp [ 2283.518398] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 21:15:11 (1788311711) [ 2284.848882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2286.127320] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:15:14 (1788311714) [ 2294.097478] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:15:22 (1788311722) [ 2299.876303] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:15:28 (1788311728) [ 2305.502625] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:15:34 (1788311734) [ 2312.346774] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:15:40 (1788311740) [ 2318.805350] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:15:47 (1788311747) [ 2326.376519] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:15:54 (1788311754) [ 2332.986555] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:16:01 (1788311761) [ 2339.755388] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:16:08 (1788311768) [ 2345.477306] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:16:14 (1788311774) [ 2350.417790] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:16:19 (1788311779) [ 2356.233794] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:16:24 (1788311784) [ 2361.536067] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:16:30 (1788311790) [ 2367.958312] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:16:36 (1788311796) [ 2374.126700] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:16:42 (1788311802) [ 2380.548688] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:16:49 (1788311809) [ 2387.060748] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:16:55 (1788311815) [ 2394.529562] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:17:03 (1788311823) [ 2394.918996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2394.946630] LNet: 136748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2394.960900] LNetError: 136748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.984367] LNet: Removed LNI 192.168.201.42@tcp2 [ 2395.035938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2395.242702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2395.283294] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2396.720493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2396.750530] LNet: 136890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.759694] LNetError: 136890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.775437] LNet: Removed LNI 192.168.201.42@tcp [ 2396.795532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.887581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2396.935580] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2404.605975] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:17:12 (1788311832) [ 2411.468556] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:17:20 (1788311840) [ 2417.605194] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:17:26 (1788311846) [ 2423.722845] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:17:32 (1788311852) [ 2430.104664] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:17:38 (1788311858) [ 2436.002031] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:17:44 (1788311864) [ 2441.876248] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:17:50 (1788311870) [ 2447.999099] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:17:56 (1788311876) [ 2455.152992] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:18:03 (1788311883) [ 2463.490939] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:18:11 (1788311891) [ 2470.709321] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:18:19 (1788311899) [ 2477.382327] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:18:26 (1788311906) [ 2483.176159] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2242 sec ========== 21:18:31 (1788311911) [ 2483.515797] LNet: 144355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2483.519587] LNetError: 144355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2483.530798] LNet: Removed LNI 192.168.201.42@tcp [ 2483.968202] Key type .llcrypt unregistered [ 2483.969948] Key type ._llcrypt unregistered [ 2493.915438] Key type ._llcrypt registered [ 2493.917954] Key type .llcrypt registered [ 2494.279073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2494.289719] alg: No test for adler32 (adler32-zlib) [ 2495.230187] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2495.454030] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2497.111667] Key type lgssc registered [ 2498.262566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2556.742927] Lustre: Mounted lustre-client - version 2.17.57_45_g07d9be3 [ 2560.971225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2567.403839] Lustre: Unmounted lustre-client [ 2588.569586] Key type lgssc unregistered [ 2588.865322] LNet: 147312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2588.874461] LNetError: 147312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2588.898881] LNet: Removed LNI 192.168.201.42@tcp [ 2589.889185] Key type .llcrypt unregistered [ 2589.890932] Key type ._llcrypt unregistered