[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 442547603 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003214] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007904] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.010124] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.021649] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026276] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] *** VALIDATE tmpfs *** [ 0.032355] *** VALIDATE proc *** [ 0.034071] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036233] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039039] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043453] debug: unmapping init [mem 0xffffffffb2859000-0xffffffffb2860fff] [ 0.045183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046719] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048017] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053017] ... event mask: 000000070000000f [ 0.054315] rcu: Hierarchical SRCU implementation. [ 0.056509] smp: Bringing up secondary CPUs ... [ 0.057655] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.062153] smp: Brought up 1 node, 4 CPUs [ 0.064021] smpboot: Max logical packages: 1 [ 0.065016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.256260] node 0 deferred pages initialised in 188ms [ 0.261235] devtmpfs: initialized [ 0.262431] x86/mm: Memory block size: 128MB [ 0.266102] gcov: version magic: 0x41383552 [ 0.269566] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.276142] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.279414] pinctrl core: initialized pinctrl subsystem [ 0.281238] [ 0.281780] ************************************************************* [ 0.284018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285000] ** ** [ 0.287014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.290019] ** ** [ 0.291015] ** This means that this kernel is built to expose internal ** [ 0.293015] ** IOMMU data structures, which may compromise security on ** [ 0.295016] ** your system. ** [ 0.297016] ** ** [ 0.299021] ** If you see this message and you are not debugging the ** [ 0.302017] ** kernel, report this immediately to your vendor! ** [ 0.304016] ** ** [ 0.307014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.308010] ************************************************************* [ 0.310991] NET: Registered protocol family 16 [ 0.313462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.315124] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.317086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.322046] cpuidle: using governor menu [ 0.324034] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.326553] PCI: Using configuration type 1 for base access [ 0.328314] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.339096] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.341043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.344125] cryptd: max_cpu_qlen set to 1000 [ 0.348032] ACPI: Added _OSI(Module Device) [ 0.350023] ACPI: Added _OSI(Processor Device) [ 0.351015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.353019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.358475] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.364494] ACPI: Interpreter enabled [ 0.366088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.368020] ACPI: Using IOAPIC for interrupt routing [ 0.370169] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.373441] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.383000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.385052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.388027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.391099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.396594] acpiphp: Slot [2] registered [ 0.398212] acpiphp: Slot [5] registered [ 0.399192] acpiphp: Slot [6] registered [ 0.401142] acpiphp: Slot [3] registered [ 0.402165] acpiphp: Slot [4] registered [ 0.404119] acpiphp: Slot [7] registered [ 0.405116] acpiphp: Slot [8] registered [ 0.407123] acpiphp: Slot [9] registered [ 0.408085] acpiphp: Slot [10] registered [ 0.409157] acpiphp: Slot [11] registered [ 0.411099] acpiphp: Slot [12] registered [ 0.412121] acpiphp: Slot [13] registered [ 0.413107] acpiphp: Slot [14] registered [ 0.414182] acpiphp: Slot [15] registered [ 0.415077] acpiphp: Slot [16] registered [ 0.417128] acpiphp: Slot [17] registered [ 0.418112] acpiphp: Slot [18] registered [ 0.420133] acpiphp: Slot [19] registered [ 0.421150] acpiphp: Slot [20] registered [ 0.423176] acpiphp: Slot [21] registered [ 0.424117] acpiphp: Slot [22] registered [ 0.426137] acpiphp: Slot [23] registered [ 0.428149] acpiphp: Slot [24] registered [ 0.429078] acpiphp: Slot [25] registered [ 0.430137] acpiphp: Slot [26] registered [ 0.432126] acpiphp: Slot [27] registered [ 0.433173] acpiphp: Slot [28] registered [ 0.434110] acpiphp: Slot [29] registered [ 0.436149] acpiphp: Slot [30] registered [ 0.438123] acpiphp: Slot [31] registered [ 0.439079] PCI host bridge to bus 0000:00 [ 0.441033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.444104] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.446029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.449041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.451044] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.454038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.455212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.458062] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.461221] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.469579] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.474066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.477026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.479021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.482039] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.485065] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.487785] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.490057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.493925] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.498020] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.507000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.511018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.516209] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.525015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.531019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.543771] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.554885] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.565019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.575023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.597009] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.614073] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.617505] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.620470] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.623507] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.626290] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.631039] iommu: Default domain type: Passthrough [ 0.633395] SCSI subsystem initialized [ 0.635161] ACPI: bus type USB registered [ 0.637101] usbcore: registered new interface driver usbfs [ 0.639111] usbcore: registered new interface driver hub [ 0.641072] usbcore: registered new device driver usb [ 0.643194] pps_core: LinuxPPS API ver. 1 registered [ 0.645014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.649071] PTP clock support registered [ 0.651216] EDAC MC: Ver: 3.0.0 [ 0.654161] PCI: Using ACPI for IRQ routing [ 0.655558] NetLabel: Initializing [ 0.657019] NetLabel: domain hash size = 128 [ 0.659018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.661142] NetLabel: unlabeled traffic allowed by default [ 0.664095] vgaarb: loaded [ 0.665303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.667017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.673012] clocksource: Switched to clocksource kvm-clock [ 0.785966] VFS: Disk quotas dquot_6.6.0 [ 0.787809] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.790554] *** VALIDATE ramfs *** [ 0.791271] *** VALIDATE hugetlbfs *** [ 0.793380] pnp: PnP ACPI init [ 0.795811] pnp: PnP ACPI: found 6 devices [ 0.814153] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.818027] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.820581] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.825169] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.827867] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.830685] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.833723] NET: Registered protocol family 2 [ 0.836222] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.840718] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.845119] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.850414] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.853881] TCP: Hash tables configured (established 65536 bind 65536) [ 0.857142] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.860334] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.863794] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.866669] NET: Registered protocol family 1 [ 0.869928] RPC: Registered named UNIX socket transport module. [ 0.872160] RPC: Registered udp transport module. [ 0.874080] RPC: Registered tcp transport module. [ 0.876098] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.878676] NET: Registered protocol family 44 [ 0.880368] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.882896] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.885362] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.887914] PCI: CLS 0 bytes, default 64 [ 0.889842] Unpacking initramfs... [ 2.284080] debug: unmapping init [mem 0xffff9a67bcc64000-0xffff9a67bffcffff] [ 2.288487] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.290980] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.294872] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.812177] Initialise system trusted keyrings [ 2.813822] Key type blacklist registered [ 2.815733] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.825304] zbud: loaded [ 2.828542] *** VALIDATE nfs *** [ 2.829917] *** VALIDATE nfs4 *** [ 2.831837] pstore: using deflate compression [ 2.836033] Platform Keyring initialized [ 2.942969] NET: Registered protocol family 38 [ 2.945287] Key type asymmetric registered [ 2.947286] Asymmetric key parser 'x509' registered [ 2.949529] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.953066] io scheduler mq-deadline registered [ 2.954889] io scheduler kyber registered [ 2.956989] io scheduler bfq registered [ 2.959314] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.962687] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.965835] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.969180] ACPI: Power Button [PWRF] [ 2.975095] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.981913] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.998049] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.025065] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.053347] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.059146] Non-volatile memory driver v1.3 [ 3.061036] Linux agpgart interface v0.103 [ 3.096841] virtio_blk virtio1: [vda] 145784 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.099824] vda: detected capacity change from 0 to 74641408 [ 3.114105] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.116824] vdb: detected capacity change from 0 to 1073741824 [ 3.122753] libphy: Fixed MDIO Bus: probed [ 3.127542] usbcore: registered new interface driver usbserial_generic [ 3.130123] usbserial: USB Serial support registered for generic [ 3.132405] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.136387] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.138184] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.142054] mousedev: PS/2 mouse device common for all mice [ 3.145678] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.149873] rtc_cmos 00:05: RTC can wake from S4 [ 3.153322] rtc_cmos 00:05: registered as rtc0 [ 3.154744] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.155841] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.161740] intel_pstate: CPU model not supported [ 3.162489] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.168134] hid: raw HID events driver (C) Jiri Kosina [ 3.170592] usbcore: registered new interface driver usbhid [ 3.172421] usbhid: USB HID core driver [ 3.173966] drop_monitor: Initializing network drop monitor service [ 3.176131] Initializing XFRM netlink socket [ 3.178178] NET: Registered protocol family 10 [ 3.180988] Segment Routing with IPv6 [ 3.182198] NET: Registered protocol family 17 [ 3.183993] mpls_gso: MPLS GSO support [ 3.189894] RAS: Correctable Errors collector initialized. [ 3.192306] AVX version of gcm_enc/dec engaged. [ 3.194214] AES CTR mode by8 optimization enabled [ 3.276246] sched_clock: Marking stable (3276203467, 0)->(4211755085, -935551618) [ 3.279921] registered taskstats version 1 [ 3.282507] Loading compiled-in X.509 certificates [ 3.284517] zswap: loaded using pool lzo/zbud [ 3.308887] Key type big_key registered [ 3.321842] Key type encrypted registered [ 3.323375] ima: No TPM chip found, activating TPM-bypass! [ 3.325542] ima: Allocated hash algorithm: sha1 [ 3.327392] ima: No architecture policies found [ 3.329441] evm: Initialising EVM extended attributes: [ 3.331277] evm: security.selinux [ 3.332756] evm: security.ima [ 3.333948] evm: security.capability [ 3.335451] evm: HMAC attrs: 0x1 [ 3.338244] rtc_cmos 00:05: setting system clock to 2026-07-13 18:13:02 UTC (1783966382) [ 3.345666] debug: unmapping init [mem 0xffffffffb3803000-0xffffffffb39fffff] [ 3.349198] debug: unmapping init [mem 0xffffffffb2582000-0xffffffffb2858fff] [ 3.358125] Write protecting the kernel read-only data: 28672k [ 3.360851] debug: unmapping init [mem 0xffffffffb0c03000-0xffffffffb0dfffff] [ 3.363704] debug: unmapping init [mem 0xffffffffb1514000-0xffffffffb15fffff] [ 3.396792] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.404494] systemd[1]: Detected virtualization kvm. [ 3.406493] systemd[1]: Detected architecture x86-64. [ 3.408616] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.446400] systemd[1]: No hostname configured. [ 3.448364] systemd[1]: Set hostname to . [ 3.450663] random: systemd: uninitialized urandom read (16 bytes read) [ 3.453374] systemd[1]: Initializing machine ID from random generator. [ 3.512894] random: ln: uninitialized urandom read (6 bytes read) [ 3.599462] random: systemd: uninitialized urandom read (16 bytes read) [ 3.602497] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.608384] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.614165] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.265940] device-mapper: uevent: version 1.0.3 [ 4.268293] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. [ 4.507922] random: fast init done Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.030589] virtio_net virtio0 ens2: renamed from eth0 [ 5.039060] scsi host0: ata_piix [ 5.048101] scsi host1: ata_piix [ 5.052677] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.054973] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.521476] dracut-initqueue[599]: RTNETLINK answers: File exists [ 9.841215] random: crng init done [ 9.842755] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.329902] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.483210] printk: systemd: 21 output lines suppressed due to ratelimiting [ 11.742321] SELinux: Disabled at runtime. [ 11.802236] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.812315] systemd[1]: Detected virtualization kvm. [ 11.814391] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.329677] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.333321] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.339638] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.343466] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.346366] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.358730] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.364117] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ 12.568294] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.886181] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.273282] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.346303] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.434954] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.456661] EDAC sbridge: Ver: 1.1.2 [ 14.668588] Key type dns_resolver registered [ 14.979611] NFS: Registering the id_resolver key type [ 14.981828] Key type id_resolver registered [ 14.983541] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg620-client login: [ 70.514339] libcfs: loading out-of-tree module taints kernel. [ 70.594750] Key type ._llcrypt registered [ 70.596536] Key type .llcrypt registered [ 71.285972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 71.296683] alg: No test for adler32 (adler32-zlib) [ 72.597445] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 73.325546] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 75.087164] Key type lgssc registered [ 76.731255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 123.047482] hrtimer: interrupt took 3022536 ns [ 201.290247] Lustre: Mounted lustre-client [ 206.670617] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.712676] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing check_logdir /tmp/testlogs/ [ 226.786201] Lustre: lustre-OST0000-osc-ffff9a6803877000: disconnect after 23s idle [ 227.062539] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing yml_node [ 232.369219] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 234.907865] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 237.507488] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 238.746644] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jul 13 14:16:56 EDT 2026 [ 254.579315] Lustre: Unmounted lustre-client [ 283.891269] Key type lgssc unregistered [ 284.254893] LNet: 8424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 284.260146] LNetError: 8424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 284.283174] LNet: Removed LNI 192.168.206.20@tcp [ 285.083235] Key type .llcrypt unregistered [ 285.087244] Key type ._llcrypt unregistered [ 307.529393] Key type ._llcrypt registered [ 307.538978] Key type .llcrypt registered [ 307.708278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 307.719293] alg: No test for adler32 (adler32-zlib) [ 308.613769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 308.688547] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 308.713522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 312.082215] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 312.205811] LNet: 9302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.215990] LNetError: 9302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.247702] LNet: Removed LNI 192.168.206.20@tcp [ 312.801502] Key type .llcrypt unregistered [ 312.804591] Key type ._llcrypt unregistered [ 317.382148] Key type ._llcrypt registered [ 317.395087] Key type .llcrypt registered [ 318.798263] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:18:16 (1783966696) [ 319.154447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 319.170823] alg: No test for adler32 (adler32-zlib) [ 320.151318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.311457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.337557] LNet: 9718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.372809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.478311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 320.515448] LNet: 9725:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 327.768362] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:18:25 (1783966705) [ 327.881469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.905786] LNet: 10295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.922608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.974341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 335.851926] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:18:32 (1783966712) [ 335.952936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.976605] LNet: 10882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.023467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.145618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 343.299225] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:18:41 (1783966721) [ 343.467487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.491380] LNet: 11471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.515502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.611774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 349.997563] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:18:47 (1783966727) [ 350.433880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.475546] LNet: 12059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.520944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.656885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 351.080250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 356.907862] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:18:55 (1783966735) [ 357.002045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.067140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.155953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 357.547593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 357.990149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 363.042613] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:19:01 (1783966741) [ 363.266339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.288800] LNet: 13247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.297354] LNet: 13247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 363.313823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.426902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 368.531113] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:19:06 (1783966746) [ 368.731683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.788404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.860459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 368.904369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 369.290792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 369.339507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 369.637578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 369.697319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 369.992423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 370.034883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 370.424336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 370.478849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 370.989786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 371.011884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 371.446889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 371.496487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 371.953886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 371.988792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 372.404341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 372.435188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 372.869680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 372.909877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 373.439400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 373.472723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 373.841613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 373.875560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 374.409947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 374.493951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 380.894504] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:19:18 (1783966758) [ 380.980610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.994736] LNet: 14507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.999412] LNet: 14507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 381.021334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.146693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 381.173861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 389.018945] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:19:26 (1783966766) [ 389.223543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.305219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.378961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 389.405470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 396.907274] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:19:34 (1783966774) [ 397.064994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.121442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.247092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 397.281400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 404.228698] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:19:41 (1783966781) [ 404.309933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.348890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.396147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 404.420886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 410.591587] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:19:48 (1783966788) [ 410.693878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.755019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.870174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 410.913714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 417.590631] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:19:55 (1783966795) [ 417.765609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.782575] LNet: 17446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.787204] LNet: 17446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 417.801512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.851390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 417.881879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 423.432891] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:20:01 (1783966801) [ 423.602899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.673124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.740292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 424.119636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 431.006706] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:20:08 (1783966808) [ 431.161963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.231073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.509568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 431.777457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 438.328191] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:20:16 (1783966816) [ 438.478870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.553349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.642443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 438.795422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 444.345225] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:20:22 (1783966822) [ 444.506991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.530760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.945626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.260292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 451.685336] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:20:29 (1783966829) [ 451.794205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.866498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.987400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.047440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 459.491527] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:20:37 (1783966837) [ 459.794714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.840937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.904780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 459.952891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.211639] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:20:45 (1783966845) [ 467.424891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.471058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.543548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 473.563141] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:20:51 (1783966851) [ 473.713094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.780824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.859493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 480.080681] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:20:58 (1783966858) [ 480.221091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.252421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.335675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 480.701094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 487.423923] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:21:04 (1783966864) [ 487.513576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.544889] LNet: 23464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 487.549886] LNet: 23464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 487.586269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.701419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 487.738887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 495.171220] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:21:13 (1783966873) [ 495.452588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.528674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.588317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 495.632811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 502.480398] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:21:20 (1783966880) [ 502.644094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.704164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.824804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 502.860931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 508.829946] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:21:26 (1783966886) [ 508.921087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.982234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.165995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 509.197368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 515.004263] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:21:33 (1783966893) [ 515.327419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.375549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.459337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 515.532391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 515.562554] LNetError: 25831:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 515.610846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 515.647755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 522.806887] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:21:40 (1783966900) [ 522.934282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.031851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.185475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.72nFNrqMxi [ 523.268952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.72nFNrqMxi [ 523.360061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.72nFNrqMxi [ 530.817173] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:21:48 (1783966908) [ 531.008323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.117075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.214598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 531.288198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 531.397713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 538.171276] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 14:21:56 (1783966916) [ 538.392766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.447396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.506735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 538.530696] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 540.498138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 541.808750] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 543.847115] Key type lgssc registered [ 545.087797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 549.173946] LNetError: 27592:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.206.20@tcp99 has no available nets [ 551.848645] Key type lgssc unregistered [ 552.232656] LNetError: 28701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 552.257202] LNet: Removed LNI 192.168.206.20@tcp [ 552.802592] Key type .llcrypt unregistered [ 552.805277] Key type ._llcrypt unregistered [ 560.708502] Key type ._llcrypt registered [ 560.714154] Key type .llcrypt registered [ 562.114790] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:22:19 (1783966939) [ 562.785225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 562.791748] alg: No test for adler32 (adler32-zlib) [ 563.766528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.871274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 567.279144] LNet: Message forwarding will be enabled in 2s [ 570.336659] LNet: Message forwarding enabled [ 570.900659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 570.938173] LNet: Message forwarding will stop in 2s to 6s [ 575.343846] LNet: Message forwarding disabled [ 575.451506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 578.100160] LNet: Message forwarding will be enabled in 2s [ 581.539712] LNet: Message forwarding enabled [ 585.899368] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:22:43 (1783966963) [ 586.052554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.092644] LNet: 30106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.230334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.283635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 594.968051] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:22:52 (1783966972) [ 595.092526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.125554] LNet: 30685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.193455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.319345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 595.380025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 595.454618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 595.525207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 595.596644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 595.642357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 595.673354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 595.709519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 595.781645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 595.815770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 595.856039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 595.876580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 595.953340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 596.007693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 596.051719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 596.105782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 596.164423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 596.221908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 596.299620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 596.321904] LNetError: 30734:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 602.496352] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:23:00 (1783966980) [ 602.780177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.799620] LNet: 31306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.818877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.913605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.406413] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:23:07 (1783966987) [ 611.520541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg620-server [ 613.145681] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:23:11 (1783966991) [ 613.321861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.351145] LNet: 32236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.375882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.448401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.494363] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 614.390842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[21-25]@tcp [ 614.413260] LNet: 32369:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 614.452806] LNetError: 32240:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.21@tcp is being used as a gateway but routing feature is not turned on [ 614.483650] LNetError: 32240:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.21@tcp has gone from up to down [ 615.468387] LNetError: 32240:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.22@tcp is being used as a gateway but routing feature is not turned on [ 615.480785] LNetError: 32240:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 616.482023] LNetError: 32240:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.22@tcp is being used as a gateway but routing feature is not turned on [ 616.504388] LNetError: 32240:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 618.542723] LNetError: 32240:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.21@tcp is being used as a gateway but routing feature is not turned on [ 618.563690] LNetError: 32240:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 621.109626] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:23:18 (1783966998) [ 622.629682] LNetError: 32240:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.21@tcp is being used as a gateway but routing feature is not turned on [ 622.650623] LNetError: 32240:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 622.942230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg620-server [ 624.915832] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:23:22 (1783967002) [ 625.047237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.097265] LNet: 33301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.106596] LNetError: 33301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.166278] LNet: Removed LNI 192.168.206.20@tcp [ 625.207821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.275030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 625.296471] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 626.220155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[21-26/2]@tcp [ 626.238529] LNet: 33422:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 626.244164] LNet: 33422:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 626.264489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[21-26/2]@tcp [ 626.287664] LNetError: 33309:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.25@tcp has gone from up to down [ 626.304428] LNetError: 33309:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 632.605960] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:23:30 (1783967010) [ 633.004317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.045111] LNetError: 34015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.074707] LNet: Removed LNI 192.168.206.20@tcp [ 633.106185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.216129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 633.273380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.324609] LNet: 34029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.330201] LNet: 34029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 633.383731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.447071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 633.534129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.567428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.617304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 633.714485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.752564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.825593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 633.901474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.979213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.059184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 634.139892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.194507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.253855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 634.333231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.359546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.392711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 634.479500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.531617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.596981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 634.683798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.732552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.807681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 634.883312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.908493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.964298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 635.026158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.053722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.116925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 635.169886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.201500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.368756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 642.737665] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:23:40 (1783967020) [ 644.825948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg620-server [ 647.069903] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:23:44 (1783967024) [ 650.027624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg620-server [ 652.854527] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:23:49 (1783967029) [ 653.035249] LNet: 35409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.048646] LNet: 35409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 654.264483] Key type .llcrypt unregistered [ 654.266722] Key type ._llcrypt unregistered [ 661.645394] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 661.889899] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 661.897448] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 661.905229] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 664.440525] Key type ._llcrypt registered [ 664.447597] Key type .llcrypt registered [ 664.655272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 664.665913] alg: No test for adler32 (adler32-zlib) [ 665.634262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.709173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 665.743709] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 666.461450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 666.478842] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 668.411203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 668.435462] LNet: Removed LNI 10.1.2.3@tcp [ 668.572183] LNet: 36135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.578794] LNetError: 36135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.595211] LNet: Removed LNI 192.168.206.20@tcp [ 669.133474] Key type .llcrypt unregistered [ 669.136214] Key type ._llcrypt unregistered [ 679.993499] Key type ._llcrypt registered [ 679.996241] Key type .llcrypt registered [ 681.075521] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:24:19 (1783967059) [ 681.620503] Key type .llcrypt unregistered [ 681.622711] Key type ._llcrypt unregistered [ 687.230367] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 687.319226] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 687.326978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 687.333141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 689.842562] Key type ._llcrypt registered [ 689.845045] Key type .llcrypt registered [ 690.036361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 690.055458] alg: No test for adler32 (adler32-zlib) [ 690.967981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.020740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.045323] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 691.631197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 691.651837] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 693.754909] LNet: 37629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.762078] LNetError: 37629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 693.767607] LNet: Removed LNI 10.1.2.3@tcp [ 693.782089] LNet: Removed LNI 192.168.206.20@tcp [ 694.254982] Key type .llcrypt unregistered [ 694.257022] Key type ._llcrypt unregistered [ 704.731962] Key type ._llcrypt registered [ 704.736397] Key type .llcrypt registered [ 706.089259] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:24:43 (1783967083) [ 706.647173] Key type .llcrypt unregistered [ 706.650095] Key type ._llcrypt unregistered [ 712.895441] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 713.044400] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 713.052200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 713.060167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 714.658386] Key type ._llcrypt registered [ 714.660987] Key type .llcrypt registered [ 714.843479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.857878] alg: No test for adler32 (adler32-zlib) [ 715.828845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.878257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 715.894187] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 717.941872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 717.994533] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 720.582917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 720.659569] LNet: Removed LNI 10.1.2.3@tcp [ 720.688167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 720.745259] LNet: Removed LNI 10.1.2.31@tcp [ 720.769696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 720.800742] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 721.515968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 721.543626] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 722.443736] LNet: 39558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.452334] LNetError: 39558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 722.468064] LNet: Removed LNI 10.1.2.31@tcp [ 722.492395] LNet: Removed LNI 10.1.2.3@tcp [ 723.160627] Key type .llcrypt unregistered [ 723.164660] Key type ._llcrypt unregistered [ 734.437299] Key type ._llcrypt registered [ 734.440190] Key type .llcrypt registered [ 735.634353] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:25:13 (1783967113) [ 736.090551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 736.116814] alg: No test for adler32 (adler32-zlib) [ 736.969845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.021951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.048711] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 737.751053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.774411] LNet: 40511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.780079] LNetError: 40511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.791963] LNet: Removed LNI 192.168.206.20@tcp [ 737.808287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.842563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.20@tcp [ 737.867328] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 738.409514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.20@tcp [ 738.437573] LNet: Removed LNI 192.168.206.20@tcp [ 738.474329] LNet: 40646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.957690] Key type .llcrypt unregistered [ 738.966917] Key type ._llcrypt unregistered [ 749.093560] Key type ._llcrypt registered [ 749.095881] Key type .llcrypt registered [ 750.246535] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:25:28 (1783967128) [ 751.694486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg620-server [ 753.054977] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:25:31 (1783967131) [ 753.379610] Key type .llcrypt unregistered [ 753.382390] Key type ._llcrypt unregistered [ 759.976884] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 760.098706] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 760.102584] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 760.110532] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 762.565337] Key type ._llcrypt registered [ 762.569642] Key type .llcrypt registered [ 762.764617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 762.776912] alg: No test for adler32 (adler32-zlib) [ 763.744271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.797105] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 764.584512] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 766.809447] LNet: 42476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.817655] LNetError: 42476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.822671] LNet: Removed LNI 192.168.206.20@tcp1 [ 766.832303] LNet: Removed LNI 10.1.2.3@tcp2 [ 767.348852] Key type .llcrypt unregistered [ 767.352084] Key type ._llcrypt unregistered [ 777.517417] Key type ._llcrypt registered [ 777.521430] Key type .llcrypt registered [ 778.886163] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:25:56 (1783967156) [ 779.325396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 779.339352] alg: No test for adler32 (adler32-zlib) [ 780.268669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.312466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.331358] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 781.102816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.139365] LNet: 43446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.145971] LNetError: 43446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.178377] LNet: Removed LNI 192.168.206.20@tcp [ 781.193568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.260829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 781.284479] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 788.135282] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:26:05 (1783967165) [ 788.284645] LNet: 44029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.308945] LNetError: 44029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.327562] LNet: Removed LNI 192.168.206.20@tcp [ 788.975588] Key type .llcrypt unregistered [ 788.981072] Key type ._llcrypt unregistered [ 795.539572] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.709921] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.716371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.724284] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 798.409103] Key type ._llcrypt registered [ 798.411444] Key type .llcrypt registered [ 798.636569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.660984] alg: No test for adler32 (adler32-zlib) [ 799.606876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.677673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.708263] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 800.410914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 800.443554] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.441788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.464780] LNet: 44777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.469251] LNetError: 44777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.474271] LNet: Removed LNI 10.1.2.3@tcp [ 802.493940] LNet: Removed LNI 192.168.206.20@tcp [ 802.508087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.556192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 802.585113] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 802.590614] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 803.062671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.094479] LNet: 44804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.106465] LNetError: 44804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 803.117811] LNetError: 44804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 803.127599] LNet: Removed LNI 10.1.2.3@tcp [ 803.137119] LNet: Removed LNI 192.168.206.20@tcp [ 803.164327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.229714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 803.263934] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 803.270382] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 803.878566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 803.923505] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 805.001580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.052969] LNet: 44966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.073974] LNetError: 44966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 805.085928] LNetError: 44966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 805.093588] LNet: Removed LNI 10.1.2.3@tcp [ 805.099471] LNet: Removed LNI 192.168.206.20@tcp [ 805.112722] LNet: Removed LNI 192.168.206.20@tcp2 [ 805.130201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.232480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 805.271264] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 805.281163] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.286121] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 812.544274] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:26:30 (1783967190) [ 812.678228] LNet: 45561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.695559] LNetError: 45561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 812.708750] LNetError: 45561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 812.730634] LNet: Removed LNI 10.1.2.3@tcp [ 812.737713] LNet: Removed LNI 192.168.206.20@tcp [ 812.773246] LNet: Removed LNI 192.168.206.20@tcp2 [ 813.311922] Key type .llcrypt unregistered [ 813.319171] Key type ._llcrypt unregistered [ 818.993475] Key type ._llcrypt registered [ 818.999451] Key type .llcrypt registered [ 819.272279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 819.318873] alg: No test for adler32 (adler32-zlib) [ 820.228384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.278344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.313733] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 821.173705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.205346] LNet: 45959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.214744] LNetError: 45959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.249427] LNet: Removed LNI 192.168.206.20@tcp [ 821.274401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.335250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 821.367179] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 821.841882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.860657] LNet: 45989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.867258] LNetError: 45989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.879282] LNet: Removed LNI 192.168.206.20@tcp [ 821.910957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.977136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 822.000779] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 822.466528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.539038] LNet: Removed LNI 192.168.206.20@tcp [ 822.558831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.642830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 822.667114] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 823.125321] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 823.208886] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 823.221679] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 823.228494] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 825.719664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.755466] LNet: 46098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.761851] LNet: 46098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 825.768557] LNetError: 46098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.786703] LNetError: 46098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 825.829846] LNet: Removed LNI 192.168.206.20@tcp [ 825.856805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.969525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 826.015075] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 833.207576] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:26:50 (1783967210) [ 833.311221] LNet: 46688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.321547] LNetError: 46688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.341178] LNet: Removed LNI 192.168.206.20@tcp [ 834.052060] Key type .llcrypt unregistered [ 834.055408] Key type ._llcrypt unregistered [ 840.171117] Key type ._llcrypt registered [ 840.173194] Key type .llcrypt registered [ 840.302449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.323076] alg: No test for adler32 (adler32-zlib) [ 841.281188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.307818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 841.322462] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 841.884916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.900583] LNet: 47093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.904379] LNetError: 47093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.920625] LNet: Removed LNI 192.168.206.20@tcp [ 841.941710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.995376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 842.009164] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 848.470391] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:27:06 (1783967226) [ 848.588255] LNet: 47677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.605341] LNetError: 47677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.639524] LNet: Removed LNI 192.168.206.20@tcp [ 849.290764] Key type .llcrypt unregistered [ 849.293423] Key type ._llcrypt unregistered [ 855.754404] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.859196] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.868816] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.877079] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 858.727972] Key type ._llcrypt registered [ 858.729716] Key type .llcrypt registered [ 858.949671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.969221] alg: No test for adler32 (adler32-zlib) [ 859.902423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.962975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.991201] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 861.337590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.375230] LNet: 48135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.387649] LNetError: 48135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.410120] LNet: Removed LNI 192.168.206.20@tcp [ 861.440778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.541982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 861.559386] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 869.004468] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:27:27 (1783967247) [ 869.384940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.410831] LNet: 48734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.418233] LNetError: 48734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.429763] LNet: Removed LNI 192.168.206.20@tcp [ 869.452730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.527397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 875.789296] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:27:33 (1783967253) [ 876.056709] LNet: 49316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.832991] Key type .llcrypt unregistered [ 876.835520] Key type ._llcrypt unregistered [ 882.974625] Key type ._llcrypt registered [ 882.976327] Key type .llcrypt registered [ 883.196461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.216167] alg: No test for adler32 (adler32-zlib) [ 884.081804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.130222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.157746] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 885.004586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.026567] LNet: 49714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.032644] LNetError: 49714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.046456] LNet: Removed LNI 192.168.206.20@tcp [ 885.085239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.158981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 885.185365] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 892.506865] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:27:50 (1783967270) [ 892.613390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.650132] LNet: 50300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.661194] LNetError: 50300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.691128] LNet: Removed LNI 192.168.206.20@tcp [ 892.747162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.822867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 899.670344] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:27:57 (1783967277) [ 901.677210] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 903.254519] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:28:01 (1783967281) [ 904.640900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 906.235975] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:28:04 (1783967284) [ 907.657953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 909.297806] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:28:07 (1783967287) [ 910.831129] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 912.248395] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:28:10 (1783967290) [ 913.906225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 915.915272] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 14:28:13 (1783967293) [ 916.006268] LNet: 52354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.676683] Key type .llcrypt unregistered [ 916.679173] Key type ._llcrypt unregistered [ 922.186923] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 922.285107] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 922.288281] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 922.291546] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 924.842830] Key type ._llcrypt registered [ 924.846500] Key type .llcrypt registered [ 925.036397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.055979] alg: No test for adler32 (adler32-zlib) [ 925.983318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.074164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 926.094089] LNet: Added LNI 192.168.206.20@tcp [8/512/0/180] [ 926.099109] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 926.101546] LNet: 52683:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 926.174739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.191895] LNet: 52702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.196604] LNetError: 52702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.201823] LNet: Removed LNI 10.1.2.3@tcp [ 926.204294] LNetError: 52702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.213210] LNet: Removed LNI 192.168.206.20@tcp [ 926.228763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.303578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 926.322160] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 926.387816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.404422] LNet: 52726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.409609] LNetError: 52726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.432422] LNet: Removed LNI 192.168.206.20@tcp [ 926.455315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.523982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 926.549632] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 926.552905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 926.554854] LNet: 52734:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 926.622262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.637261] LNet: 52752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.642801] LNetError: 52752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.649358] LNet: Removed LNI 10.1.2.3@tcp [ 926.651616] LNetError: 52752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.667349] LNet: Removed LNI 192.168.206.20@tcp [ 926.690953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.740407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 926.757153] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 926.767237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 926.769379] LNet: 52760:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 926.941414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.962232] LNet: 52786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.967811] LNetError: 52786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.971621] LNet: Removed LNI 10.1.2.3@tcp [ 926.973840] LNetError: 52786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.989900] LNet: Removed LNI 192.168.206.20@tcp [ 927.024578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.120542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 927.141811] LNet: Added LNI 192.168.206.20@tcp [8/256/0/182] [ 927.146884] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 927.237762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.257141] LNet: 52814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.262572] LNetError: 52814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 927.266612] LNet: Removed LNI 10.1.2.3@tcp [ 927.271802] LNetError: 52814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.294238] LNet: Removed LNI 192.168.206.20@tcp [ 927.322254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.406761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 927.437711] LNet: Added LNI 192.168.206.20@tcp [8/256/0/182] [ 927.445744] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 927.450209] LNet: 52822:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 927.554219] LNet: Removed LNI 10.1.2.3@tcp [ 927.571756] LNet: Removed LNI 192.168.206.20@tcp [ 928.124316] Key type .llcrypt unregistered [ 928.127249] Key type ._llcrypt unregistered [ 938.749184] Key type ._llcrypt registered [ 938.751903] Key type .llcrypt registered [ 940.021826] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:28:37 (1783967317) [ 940.525139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.548312] alg: No test for adler32 (adler32-zlib) [ 941.534680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.618330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.647571] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 942.321557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.20 [ 942.353962] LNet: 53798:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 942.366393] LNet: Removed LNI 192.168.206.20@tcp [ 948.127707] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:28:45 (1783967325) [ 948.189424] LNet: 54360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.193889] LNetError: 54360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.223051] LNet: Removed LNI 192.168.206.20@tcp [ 948.984962] Key type .llcrypt unregistered [ 948.988407] Key type ._llcrypt unregistered [ 954.878852] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 955.146608] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 957.966813] Key type ._llcrypt registered [ 957.970177] Key type .llcrypt registered [ 958.125749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 958.142512] alg: No test for adler32 (adler32-zlib) [ 959.505390] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.839736] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:29:03 (1783967343) [ 966.099616] LNet: 55281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.104697] LNetError: 55281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 966.120538] LNet: Removed LNI 10.1.2.3@tcp [ 966.593639] Key type .llcrypt unregistered [ 966.600564] Key type ._llcrypt unregistered [ 972.261729] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 972.454123] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 972.458723] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 972.464595] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 975.237656] Key type ._llcrypt registered [ 975.239810] Key type .llcrypt registered [ 975.391556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.412302] alg: No test for adler32 (adler32-zlib) [ 976.354283] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 982.054715] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:29:19 (1783967359) [ 983.202437] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 984.663290] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:29:22 (1783967362) [ 984.898989] LNet: 56466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.904733] LNetError: 56466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 984.926232] LNet: Removed LNI 10.1.2.3@tcp [ 985.553944] Key type .llcrypt unregistered [ 985.556491] Key type ._llcrypt unregistered [ 991.608713] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 991.780301] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 991.782739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 991.787713] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 994.392890] Key type ._llcrypt registered [ 994.394823] Key type .llcrypt registered [ 994.648747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 994.683247] alg: No test for adler32 (adler32-zlib) [ 995.631379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.775366] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1002.972413] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:29:41 (1783967381) [ 1003.623645] LNet: 57695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.630692] LNetError: 57695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.648198] LNet: Removed LNI 10.1.2.3@tcp [ 1004.169459] Key type .llcrypt unregistered [ 1004.171855] Key type ._llcrypt unregistered [ 1010.725439] Key type ._llcrypt registered [ 1010.728752] Key type .llcrypt registered [ 1011.036104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.066957] alg: No test for adler32 (adler32-zlib) [ 1011.986592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1012.029546] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1018.755571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1019.503403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_interrupt -r 1 [ 1019.550939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_interrupt -r 1 [ 1019.623574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1019.642850] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1019.902082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.944987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1020.002903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.026928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.059749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.090639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.130973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.161261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.182298] LNet: 58133:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1020.205711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.236513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.256454] LNet: 1 local NIs in recovery (showing 1): 192.168.206.20@tcp [ 1020.262980] LNet: There was an unexpected network error while writing to 192.168.206.20: rc = -22 [ 1020.274758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.339655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.365804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1020.412806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1020.476459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1021.007170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_dropped -r 1 [ 1021.037315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_dropped -r 1 [ 1021.074513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1021.285419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1021.315408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1021.349404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1021.395091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1021.456625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1021.488547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1021.529552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1021.569857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1021.593551] LNet: 58242:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1021.613277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1021.655722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1021.705807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1021.765437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1021.813279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.855748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1021.902913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1022.407788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_aborted -r 1 [ 1022.454572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_aborted -r 1 [ 1022.508106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1022.538352] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1022.551877] LNet: Skipped 1 previous similar message [ 1022.798312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1022.825495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1022.854487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1022.901737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1022.927510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1022.952301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1022.983711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.014872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.046803] LNet: 58351:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.074175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.109965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.163329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.229729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.299710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.351442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.390431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1023.862859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_no_route -r 1 [ 1023.911880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_no_route -r 1 [ 1023.964387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1024.221786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.252759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1024.295988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1024.344378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1024.395244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1024.430321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1024.469469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1024.519639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1024.561182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1024.593988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.626185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.650167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.677593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.721691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1024.769932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1025.229414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_timeout -r 1 [ 1025.259991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_timeout -r 1 [ 1025.300589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1025.323949] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1025.329520] LNet: Skipped 2 previous similar messages [ 1025.499924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1025.528291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1025.551692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.571296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.598191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.622589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.646940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.670216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.682921] LNet: 58570:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1025.691985] LNet: 58570:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1025.709543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.731180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.753193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.776444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.807927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.834479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.856917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1026.359698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_error -r 1 [ 1026.411863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_error -r 1 [ 1026.452722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1026.668593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.716466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1026.755789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.814511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.861514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.937907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.981787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1027.020909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1027.054588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.089830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.144170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.173832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.206846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.263728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1027.317249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1027.566308] LNet: 58720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.575535] LNetError: 58720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.591587] LNet: Removed LNI 192.168.206.20@tcp [ 1027.979541] Key type .llcrypt unregistered [ 1027.983300] Key type ._llcrypt unregistered [ 1044.751408] Key type ._llcrypt registered [ 1044.755598] Key type .llcrypt registered [ 1046.555812] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:30:23 (1783967423) [ 1047.274281] Key type .llcrypt unregistered [ 1047.276301] Key type ._llcrypt unregistered [ 1054.458295] Key type ._llcrypt registered [ 1054.461195] Key type .llcrypt registered [ 1054.632118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.640401] alg: No test for adler32 (adler32-zlib) [ 1055.570987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1055.613974] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1062.842878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1072.064573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.112439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1072.135235] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1073.314866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_interrupt -r 1 [ 1073.354854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_interrupt -r 1 [ 1073.414737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_interrupt -r 1 [ 1073.487596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_interrupt -r 1 [ 1073.536160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_interrupt -r 1 [ 1073.574620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_interrupt -r 1 [ 1073.627874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_interrupt -r 1 [ 1073.675827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_interrupt -r 1 [ 1073.726873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1073.756306] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1073.773337] LNet: 1 local NIs in recovery (showing 1): 192.168.206.20@tcp [ 1074.001371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1074.017629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1074.046471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1074.079603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1074.126902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.152280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.174958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.211763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.225612] LNet: 60187:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.245242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.275772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.306858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.339321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.393626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.457859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.503970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1074.966664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_dropped -r 1 [ 1075.012576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_dropped -r 1 [ 1075.062877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_dropped -r 1 [ 1075.100843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_dropped -r 1 [ 1075.131301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_dropped -r 1 [ 1075.159387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_dropped -r 1 [ 1075.186752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_dropped -r 1 [ 1075.216857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_dropped -r 1 [ 1075.263835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1075.278977] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1075.288987] LNet: Skipped 2 previous similar messages [ 1075.567600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.586270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1075.611072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.642918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.675504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.699927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.729264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.752853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.775744] LNet: 60308:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.788582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.809823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.827237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.846326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.878689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.900437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.934666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1076.415070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_aborted -r 1 [ 1076.445661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_aborted -r 1 [ 1076.480689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_aborted -r 1 [ 1076.525724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_aborted -r 1 [ 1076.566147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_aborted -r 1 [ 1076.609251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_aborted -r 1 [ 1076.655157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_aborted -r 1 [ 1076.691523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_aborted -r 1 [ 1076.721422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1076.737434] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1076.741105] LNet: Skipped 2 previous similar messages [ 1076.962769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.999433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1077.031521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1077.069387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1077.101338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1077.135631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1077.168334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1077.214282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.229821] LNet: 60431:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1077.255206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1077.278765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.328401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.360974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.399389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.435805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.508517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1078.073169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_no_route -r 1 [ 1078.118567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_no_route -r 1 [ 1078.169357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_no_route -r 1 [ 1078.208181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_no_route -r 1 [ 1078.280170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_no_route -r 1 [ 1078.357647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_no_route -r 1 [ 1078.406969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_no_route -r 1 [ 1078.451256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_no_route -r 1 [ 1078.497649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1078.711585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1078.750963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1078.787607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1078.827423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1078.860490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1078.884630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1078.907872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1078.934656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1078.955849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.995856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.021787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.051821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.087354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.121433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.167153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1079.655481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_timeout -r 1 [ 1079.732442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_timeout -r 1 [ 1079.778905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_timeout -r 1 [ 1079.824659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_timeout -r 1 [ 1079.859716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_timeout -r 1 [ 1079.901652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_timeout -r 1 [ 1079.947372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_timeout -r 1 [ 1080.007232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_timeout -r 1 [ 1080.050914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1080.079566] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1080.086372] LNet: Skipped 5 previous similar messages [ 1080.311707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1080.330915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1080.359568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.396497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1080.429502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.465538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.490352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.546463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1080.576920] LNet: 60673:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1080.587390] LNet: 60673:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1080.613353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1080.643210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1080.674401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1080.708520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1080.773945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1080.823082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1080.885948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1081.443198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_error -r 1 [ 1081.490790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_error -r 1 [ 1081.533526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_error -r 1 [ 1081.575220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_error -r 1 [ 1081.641446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_error -r 1 [ 1081.682783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_error -r 1 [ 1081.733554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_error -r 1 [ 1081.768875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_error -r 1 [ 1081.801789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1081.998389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.020380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1082.039435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.062122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.089581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.130673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1082.177815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1082.220815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1082.255501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1082.291859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1082.326538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.369835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.413616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.460692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.503683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1087.629273] LNet: 60870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.646293] LNetError: 60870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.657542] LNet: Removed LNI 192.168.206.20@tcp [ 1088.677940] LNet: Removed LNI 192.168.206.20@tcp1 [ 1089.143656] Key type .llcrypt unregistered [ 1089.146085] Key type ._llcrypt unregistered [ 1104.833553] Key type ._llcrypt registered [ 1104.836430] Key type .llcrypt registered [ 1106.117701] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:31:23 (1783967483) [ 1107.138368] Key type .llcrypt unregistered [ 1107.140297] Key type ._llcrypt unregistered [ 1113.121570] Key type ._llcrypt registered [ 1113.124588] Key type .llcrypt registered [ 1113.332561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1113.346799] alg: No test for adler32 (adler32-zlib) [ 1114.301929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1114.345393] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1121.506617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1122.174561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e remote_dropped -r 1 [ 1122.207735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e remote_dropped -r 1 [ 1122.233644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1122.248614] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1122.415860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.450446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.487121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.519451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.544329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.565223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.587600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.603873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.616384] LNet: 62118:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.634374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.666900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.691539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.722851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.757826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.800417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.873981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1123.466797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e remote_error -r 1 [ 1123.502920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e remote_error -r 1 [ 1123.546092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1123.582828] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1123.806908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.870862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1123.923328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.984133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.024399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.053804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.089736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.147711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.167700] LNet: 62228:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.184641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.210587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.239609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.264786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.292297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.319684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.353565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1124.855872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e remote_timeout -r 1 [ 1124.887288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e remote_timeout -r 1 [ 1124.914920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1124.934455] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1125.150601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.185297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.222659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.260178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.294794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.324719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.361876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.392662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.411423] LNet: 62337:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1125.424665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.453415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.488835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.525446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.568496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.612624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.648436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1125.816800] LNet: 62378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.831396] LNetError: 62378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.852354] LNet: Removed LNI 192.168.206.20@tcp [ 1126.197489] Key type .llcrypt unregistered [ 1126.201512] Key type ._llcrypt unregistered [ 1141.206245] Key type ._llcrypt registered [ 1141.208903] Key type .llcrypt registered [ 1142.373600] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:32:00 (1783967520) [ 1143.110508] Key type .llcrypt unregistered [ 1143.114868] Key type ._llcrypt unregistered [ 1148.713598] Key type ._llcrypt registered [ 1148.717973] Key type .llcrypt registered [ 1148.903373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1148.918870] alg: No test for adler32 (adler32-zlib) [ 1149.865853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1149.917426] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1156.493859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1165.265150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.327310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1165.362171] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1166.836942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e remote_dropped -r 1 [ 1166.874519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e remote_dropped -r 1 [ 1166.934425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e remote_dropped -r 1 [ 1166.980524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e remote_dropped -r 1 [ 1167.025246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e remote_dropped -r 1 [ 1167.078227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e remote_dropped -r 1 [ 1167.121390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e remote_dropped -r 1 [ 1167.150894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e remote_dropped -r 1 [ 1167.171163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1167.185980] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1167.190349] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.120@tcp [ 1167.397580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.428548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.462398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.507600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.538019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.578219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.607858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.663560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.684731] LNet: 63844:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.705477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.747887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.788882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.817550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.844606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.898659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.938867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.984295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.661609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e remote_error -r 1 [ 1168.728846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e remote_error -r 1 [ 1168.788140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e remote_error -r 1 [ 1168.855303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e remote_error -r 1 [ 1168.939492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e remote_error -r 1 [ 1168.979110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e remote_error -r 1 [ 1169.028798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e remote_error -r 1 [ 1169.063874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e remote_error -r 1 [ 1169.102620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1169.122271] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1169.131044] LNet: Skipped 2 previous similar messages [ 1169.355867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.391691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.447989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.492695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.514420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.535628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.563610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.588925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.604685] LNet: 63967:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.628149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.662365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.690190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.717609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.746588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.799684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.841495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.880766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.473311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e remote_timeout -r 1 [ 1170.508273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e remote_timeout -r 1 [ 1170.541530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e remote_timeout -r 1 [ 1170.573727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e remote_timeout -r 1 [ 1170.599194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e remote_timeout -r 1 [ 1170.622656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e remote_timeout -r 1 [ 1170.662477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e remote_timeout -r 1 [ 1170.709941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e remote_timeout -r 1 [ 1170.741314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.120@tcp [ 1170.778185] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1170.944776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.968919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.990859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.013557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.035561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.063930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.089834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.115413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.129605] LNet: 64091:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.147936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.176307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.223376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.263954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.307266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.337204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.384090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1171.420613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.704470] LNet: 64169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.714839] LNetError: 64169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.719848] LNet: Removed LNI 192.168.206.20@tcp [ 1175.732580] LNet: Removed LNI 192.168.206.20@tcp1 [ 1176.096647] Key type .llcrypt unregistered [ 1176.099169] Key type ._llcrypt unregistered [ 1192.868330] Key type ._llcrypt registered [ 1192.870982] Key type .llcrypt registered [ 1194.233571] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:32:51 (1783967571) [ 1195.644653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1197.074598] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:32:55 (1783967575) [ 1197.936358] Key type .llcrypt unregistered [ 1197.942530] Key type ._llcrypt unregistered [ 1204.009549] Key type ._llcrypt registered [ 1204.012432] Key type .llcrypt registered [ 1204.166366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1204.182577] alg: No test for adler32 (adler32-zlib) [ 1205.077429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1205.120561] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1211.922630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1212.529370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e network_timeout -r 1 [ 1212.567344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e network_timeout -r 1 [ 1212.596416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1212.610370] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1212.789109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.818137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1212.843223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.877232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.899230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.923190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.947521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.978894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.995986] LNet: 65713:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.016244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.039639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.077696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.102865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.127669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.149632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.380351] LNet: 65752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.393333] LNetError: 65752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.416705] LNet: Removed LNI 192.168.206.20@tcp [ 1213.779623] Key type .llcrypt unregistered [ 1213.782062] Key type ._llcrypt unregistered [ 1226.931814] Key type ._llcrypt registered [ 1226.936644] Key type .llcrypt registered [ 1227.112793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.125712] alg: No test for adler32 (adler32-zlib) [ 1228.069029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1228.093093] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1234.814730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1243.115721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.189835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1243.234306] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1244.422878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e network_timeout -r 1 [ 1244.464816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e network_timeout -r 1 [ 1244.494209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e network_timeout -r 1 [ 1244.520565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e network_timeout -r 1 [ 1244.561409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e network_timeout -r 1 [ 1244.595602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e network_timeout -r 1 [ 1244.619543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e network_timeout -r 1 [ 1244.672330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e network_timeout -r 1 [ 1244.734524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1244.760793] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1244.966351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.997079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.038243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.074286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.143855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.190663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.223591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.251993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.267084] LNet: 66632:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.285467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.313212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.340184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.365359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.409866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.471576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.286274] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1250.492778] LNet: 66706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.497739] LNetError: 66706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.503736] LNet: Removed LNI 192.168.206.20@tcp [ 1250.518296] LNet: Removed LNI 192.168.206.20@tcp1 [ 1250.796838] Key type .llcrypt unregistered [ 1250.800783] Key type ._llcrypt unregistered [ 1265.432445] Key type ._llcrypt registered [ 1265.434441] Key type .llcrypt registered [ 1266.592298] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:34:04 (1783967644) [ 1267.229280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1267.245914] alg: No test for adler32 (adler32-zlib) [ 1268.165067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1268.220261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1268.240709] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1268.887990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1268.911691] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1269.535594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 1269.591816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1269.631790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1269.678740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1269.724197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1269.765698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1269.800340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.20@tcp [ 1269.827844] LNetError: 67806:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.206.20@tcp to recovery [ 1270.239733] LNet: 1 local NIs in recovery (showing 1): 192.168.206.20@tcp [ 1270.239885] LNet: There was an unexpected network error while writing to 192.168.206.20: rc = -22 [ 1272.287896] LNet: There was an unexpected network error while writing to 192.168.206.20: rc = -22 [ 1273.205933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1276.383395] LNet: There was an unexpected network error while writing to 192.168.206.20: rc = -22 [ 1276.688880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1276.711075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1276.785293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1276.808376] LNet: 67913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1276.815282] LNetError: 67913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1276.825378] LNet: Removed LNI 192.168.206.20@tcp [ 1276.841611] LNet: Removed LNI 192.168.206.20@tcp1 [ 1276.866731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1276.915556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1276.970084] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1277.750979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1277.766604] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1278.451578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 1278.504665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.547351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1278.575451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1278.609113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1278.635889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1278.658035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1278.683764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.20@tcp [ 1278.697139] LNetError: 68186:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.206.20@tcp to recovery [ 1278.943523] LNet: There was an unexpected network error while writing to 192.168.206.20: rc = -22 [ 1282.116875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1284.064807] LNet: There was an unexpected network error while writing to 192.168.206.20: rc = -22 [ 1284.074213] LNet: Skipped 1 previous similar message [ 1288.987041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1289.018441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1289.041582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.150354] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:34:33 (1783967673) [ 1296.244831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1296.269608] LNet: 68889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.277420] LNetError: 68889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.283668] LNetError: 68889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1296.289819] LNet: Removed LNI 192.168.206.20@tcp [ 1296.307391] LNet: Removed LNI 192.168.206.20@tcp1 [ 1296.351065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.439801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1296.493977] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1297.286781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1297.329417] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1298.080808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 1298.143400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1298.169512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1298.198673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1298.233915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1298.281733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1298.315655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.20@tcp --health 0 [ 1298.400446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1298.465277] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.20@tcp [ 1299.491078] LNet: There was an unexpected network error while writing to 192.168.206.20: rc = -22 [ 1299.496980] LNet: Skipped 1 previous similar message [ 1314.777328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1314.916138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1314.955450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1314.998817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.20@tcp --health 500 [ 1315.106836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1322.781892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1322.833489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1322.853465] LNet: 69405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.858756] LNetError: 69405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.861932] LNetError: 69405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1322.866823] LNet: Removed LNI 192.168.206.20@tcp [ 1322.878717] LNet: Removed LNI 192.168.206.20@tcp1 [ 1322.899610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1322.963859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1322.989567] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1323.576478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1323.608090] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1324.295524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 1324.357528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1324.432716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1324.452553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1324.488474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1324.513488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1324.537445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1324.559822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.20@tcp --health 0 [ 1325.983626] LNet: There was an unexpected network error while writing to 192.168.206.20: rc = -22 [ 1325.994030] LNet: Skipped 3 previous similar messages [ 1326.971287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.079400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.114397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1336.142996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1342.074750] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:35:19 (1783967719) [ 1342.441947] LNet: 70450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.450556] LNetError: 70450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1342.455399] LNetError: 70450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1342.461581] LNet: Removed LNI 192.168.206.20@tcp [ 1342.481798] LNet: Removed LNI 192.168.206.20@tcp1 [ 1343.192132] Key type .llcrypt unregistered [ 1343.196386] Key type ._llcrypt unregistered [ 1349.089337] Key type ._llcrypt registered [ 1349.092376] Key type .llcrypt registered [ 1349.255487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1349.268462] alg: No test for adler32 (adler32-zlib) [ 1350.163608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1350.203085] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1358.576536] Lustre: DEBUG MARKER: Initial discovery [ 1358.835370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.120@tcp [ 1361.057134] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1361.184641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1361.224314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.120@tcp [ 1361.266875] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1361.287817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1361.310481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.120@tcp -r 1 -m GET -l 3 [ 1361.334444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.120@tcp -r 1 -m GET -e local_error [ 1361.360253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.120@tcp -r 1 -m PUT -l 6 [ 1361.504090] LNet: 1 local NIs in recovery (showing 1): 192.168.206.20@tcp [ 1362.342462] Lustre: DEBUG MARKER: Force oleg620-client.virtnet to discover 192.168.206.120@tcp (in background) [ 1362.526677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.120@tcp [ 1363.624907] Lustre: DEBUG MARKER: Force oleg620-server to discover 192.168.206.20@tcp [ 1365.606222] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1366.000483] Lustre: DEBUG MARKER: Wait for 71215 [ 1367.211697] Lustre: DEBUG MARKER: Finished wait on 71215 [ 1367.358805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1367.403148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1367.407586] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1367.648465] LNet: 71663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.653372] LNetError: 71663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.668645] LNet: Removed LNI 192.168.206.20@tcp [ 1368.150307] Key type .llcrypt unregistered [ 1368.156627] Key type ._llcrypt unregistered [ 1383.727151] Key type ._llcrypt registered [ 1383.729193] Key type .llcrypt registered [ 1385.256882] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:36:02 (1783967762) [ 1385.888505] Key type .llcrypt unregistered [ 1385.890236] Key type ._llcrypt unregistered [ 1391.887867] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1392.038691] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1392.041476] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1392.044993] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1394.662909] Key type ._llcrypt registered [ 1394.664588] Key type .llcrypt registered [ 1394.825706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1394.850106] alg: No test for adler32 (adler32-zlib) [ 1395.812059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.867648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1395.910749] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1396.679236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1396.703306] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1405.637356] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:36:23 (1783967783) [ 1405.959754] LNet: 73742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1405.964656] LNetError: 73742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1405.971832] LNet: Removed LNI 10.1.2.3@tcp [ 1405.979992] LNet: Removed LNI 192.168.206.20@tcp [ 1406.621563] Key type .llcrypt unregistered [ 1406.627276] Key type ._llcrypt unregistered [ 1413.913665] Key type ._llcrypt registered [ 1413.916986] Key type .llcrypt registered [ 1414.057324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1414.066692] alg: No test for adler32 (adler32-zlib) [ 1415.047716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.103942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1415.126054] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1416.215990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1416.254543] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1417.402336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.20@tcp [ 1418.464270] LNetError: Unexpected error -22 connecting to 192.168.206.20@tcp at host 192.168.206.20:988 [ 1423.921096] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:36:41 (1783967801) [ 1424.089371] LNet: 74869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1424.098190] LNetError: 74869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1424.116877] LNet: Removed LNI 10.1.2.3@tcp [ 1424.148453] LNet: Removed LNI 192.168.206.20@tcp [ 1424.956539] Key type .llcrypt unregistered [ 1424.960181] Key type ._llcrypt unregistered [ 1430.469768] Key type ._llcrypt registered [ 1430.473524] Key type .llcrypt registered [ 1430.756534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1430.785889] alg: No test for adler32 (adler32-zlib) [ 1431.829719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.912590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1431.929792] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1432.823246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1432.843861] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 1433.769314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.20@tcp1 --nid 192.168.206.20@tcp2 [ 1437.554894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.20@tcp1 192.168.206.20@tcp2 [ 1437.583237] LNetError: 75708:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.20@tcp1 to 192.168.206.20@tcp2 - Route table may be misconfigured [ 1437.605680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.20@tcp2 192.168.206.20@tcp1 [ 1443.777823] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:37:01 (1783967821) [ 1444.119745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1444.157422] LNet: 76282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.166118] LNetError: 76282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.177391] LNet: Removed LNI 192.168.206.20@tcp1 [ 1445.227028] LNet: Removed LNI 192.168.206.20@tcp2 [ 1445.272272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1445.344842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1445.372054] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1446.050752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1446.072862] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1446.871491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 1446.909909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.20@tcp 192.168.206.20@tcp [ 1446.945365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.20@tcp1 192.168.206.20@tcp1 [ 1446.996339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.20@tcp -d 192.168.206.20@tcp -e network_timeout [ 1447.039690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -e network_timeout [ 1447.081109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -e network_timeout [ 1447.112587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -e network_timeout [ 1447.151732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1447.167904] LNet: There was an unexpected network error while writing to 192.168.206.20: rc = -22 [ 1447.192106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1447.286884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1447.362961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1447.391125] LNet: 1 local NIs in recovery (showing 1): 192.168.206.20@tcp [ 1452.776162] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:37:10 (1783967830) [ 1452.863564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.899646] LNet: 77138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.923642] LNetError: 77138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.935038] LNetError: 77138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1453.984186] LNet: Removed LNI 192.168.206.20@tcp [ 1453.992172] LNetError: 77138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.037762] LNet: Removed LNI 192.168.206.20@tcp1 [ 1454.067755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.136736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1454.371453] LNet: 77181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.756061] Key type .llcrypt unregistered [ 1454.758876] Key type ._llcrypt unregistered [ 1464.374305] Key type ._llcrypt registered [ 1464.376290] Key type .llcrypt registered [ 1465.593485] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:37:23 (1783967843) [ 1465.922710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1465.944445] alg: No test for adler32 (adler32-zlib) [ 1466.819194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.088511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1469.663715] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:37:27 (1783967847) [ 1469.729638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.742632] LNet: 78271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.763530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.862975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.897569] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1470.600696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1470.621654] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1471.339530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1471.362806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp1 [ 1471.393843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp1 [ 1476.896409] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:37:34 (1783967854) [ 1478.532615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg620-server [ 1480.360272] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:37:38 (1783967858) [ 1482.121657] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg620-server [ 1483.505831] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:37:41 (1783967861) [ 1485.286151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg620-server [ 1487.020832] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:37:44 (1783967864) [ 1488.825426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg620-server [ 1490.600878] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:37:48 (1783967868) [ 1492.855370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg620-server [ 1494.949598] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:37:52 (1783967872) [ 1496.674731] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg620-server [ 1498.494144] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:37:56 (1783967876) [ 1500.149897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg620-server [ 1501.620642] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:37:59 (1783967879) [ 1503.183542] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg620-server [ 1504.838317] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:38:02 (1783967882) [ 1506.513196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg620-server [ 1508.607948] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:38:06 (1783967886) [ 1508.777113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.814035] LNet: 82156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.821777] LNetError: 82156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.828688] LNet: Removed LNI 192.168.206.20@tcp [ 1509.863843] LNet: Removed LNI 192.168.206.20@tcp1 [ 1509.941947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.054147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1510.082335] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1510.872144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1510.998497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1511.244890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.286949] LNet: 82303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.293633] LNetError: 82303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.299181] LNetError: 82303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1511.315984] LNet: Removed LNI 192.168.206.20@tcp [ 1511.344911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.408656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1511.443668] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1512.418983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1512.638545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1512.663665] LNetError: 82448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.689424] LNet: Removed LNI 192.168.206.20@tcp [ 1512.721529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.780475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1512.815255] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1513.796046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1513.865469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1514.115142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1514.161529] LNet: 82596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.170855] LNet: 82596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1515.244861] LNet: Removed LNI 192.168.206.20@tcp [ 1515.300140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.378133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1515.419338] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1516.192611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1516.427746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.446885] LNetError: 82741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.455383] LNetError: 82741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1516.472705] LNet: Removed LNI 192.168.206.20@tcp [ 1516.489554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.543989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1516.592790] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1517.352418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1517.431217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1517.624482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.693555] LNet: Removed LNI 192.168.206.20@tcp [ 1518.726068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.792198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1518.830824] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1519.376410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1519.636423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.662368] LNet: 83034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.669439] LNet: 83034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1520.749523] LNet: Removed LNI 192.168.206.20@tcp [ 1520.805065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.884837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1520.926190] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1521.666718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1521.762672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1522.022115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1522.046079] LNetError: 83181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.056989] LNetError: 83181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1523.116862] LNet: Removed LNI 192.168.206.20@tcp [ 1523.183309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.359890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1523.386795] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1524.290700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1524.599235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.669535] LNet: Removed LNI 192.168.206.20@tcp [ 1525.704214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.762591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.787856] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1526.489238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1526.562442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1526.834578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.911929] LNet: Removed LNI 192.168.206.20@tcp [ 1527.937605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.008490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1528.033451] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1528.648366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1528.939869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.967944] LNet: 83619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.974600] LNet: 83619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1530.028607] LNet: Removed LNI 192.168.206.20@tcp [ 1530.072706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.138660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.180263] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1530.758269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1530.837590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1531.120733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.156614] LNetError: 83767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1531.163945] LNetError: 83767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1531.197711] LNet: Removed LNI 192.168.206.20@tcp [ 1531.230816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.283416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1531.314885] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1532.025325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1533.551716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.635415] LNet: Removed LNI 192.168.206.20@tcp [ 1534.675169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.756707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.794211] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1535.539704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1535.608611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.662044] LNet: Removed LNI 192.168.206.20@tcp [ 1535.698524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.793809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.825879] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1536.765948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1536.836241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.873249] LNet: Removed LNI 192.168.206.20@tcp [ 1536.892638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.962796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1537.050514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.087749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.147472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1537.232302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.259527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.323347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1537.401750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.450082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.505313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1537.544457] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1538.116990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.150234] LNet: Removed LNI 192.168.206.20@tcp [ 1538.177119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.228636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1543.645767] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:38:41 (1783967921) [ 1543.851502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.887123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.973662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1544.009303] LNet: Added LNI 192.168.206.20@tcp231 [8/256/0/180] [ 1545.204953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.230866] LNet: 85081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.235161] LNet: 85081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1545.263790] LNet: Removed LNI 192.168.206.20@tcp231 [ 1545.281559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.383514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1545.411989] LNet: Added LNI 192.168.206.20@tcp231 [8/256/0/180] [ 1545.416445] LNet: 85089:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1545.807590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1545.842667] LNet: Removed LNI 192.168.206.20@tcp231 [ 1545.855210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1545.883810] LNet: Added LNI 192.168.206.20@tcp231 [8/256/0/180] [ 1546.841166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.873277] LNet: Removed LNI 192.168.206.20@tcp231 [ 1546.888837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.948334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1546.976389] LNet: Added LNI 192.168.206.20@tcp231 [8/256/0/0] [ 1546.979935] LNet: 85241:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1547.391586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1547.417668] LNet: Removed LNI 192.168.206.20@tcp231 [ 1547.447350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1547.470750] LNet: Added LNI 192.168.206.20@tcp231 [8/256/0/0] [ 1548.477960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.506341] LNetError: 85384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.510737] LNetError: 85384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1548.526468] LNet: Removed LNI 192.168.206.20@tcp231 [ 1548.551234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.617549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1548.664422] LNet: Added LNI 192.168.206.20@tcp231 [8/256/0/60] [ 1548.671838] LNet: 85392:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1549.017515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1549.046978] LNet: Removed LNI 192.168.206.20@tcp231 [ 1549.070839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1549.095909] LNet: Added LNI 192.168.206.20@tcp231 [8/256/0/60] [ 1550.166887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.192559] LNet: Removed LNI 192.168.206.20@tcp231 [ 1550.210770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.284249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1550.304753] LNet: Added LNI 192.168.206.20@tcp231 [8/256/0/180] [ 1550.712879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1550.754297] LNet: Removed LNI 192.168.206.20@tcp231 [ 1550.766823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1550.791377] LNet: Added LNI 192.168.206.20@tcp231 [8/256/0/180] [ 1557.537689] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:38:55 (1783967935) [ 1557.732942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.765447] LNet: Removed LNI 192.168.206.20@tcp231 [ 1557.787291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.839472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1557.857178] LNet: Added LNI 192.168.206.20@tcp232 [8/256/0/180] [ 1558.584992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1558.624973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1558.658736] LNet: Removed LNI 192.168.206.20@tcp232 [ 1558.670592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1558.700543] LNet: Added LNI 192.168.206.20@tcp232 [8/256/0/180] [ 1559.349936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1559.381842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1559.408868] LNet: Removed LNI 192.168.206.20@tcp232 [ 1559.423698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1559.471280] LNet: Added LNI 192.168.206.20@tcp232 [8/256/0/180] [ 1560.224844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1560.283709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1560.338239] LNet: Removed LNI 192.168.206.20@tcp232 [ 1560.365074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1560.403498] LNet: Added LNI 192.168.206.20@tcp232 [8/256/0/180] [ 1561.035863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1561.085713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1561.132884] LNet: Removed LNI 192.168.206.20@tcp232 [ 1561.148715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1566.686935] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:39:04 (1783967944) [ 1567.747386] Key type .llcrypt unregistered [ 1567.748670] Key type ._llcrypt unregistered [ 1573.262795] Key type ._llcrypt registered [ 1573.266175] Key type .llcrypt registered [ 1573.489282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1573.520167] alg: No test for adler32 (adler32-zlib) [ 1574.464887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1574.505823] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1581.461614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1589.465751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.530312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1589.569020] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1591.132909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_interrupt -r 2 [ 1591.186498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_interrupt -r 2 [ 1591.241870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_interrupt -r 2 [ 1591.296841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_interrupt -r 2 [ 1591.335349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_interrupt -r 2 [ 1591.359906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_interrupt -r 2 [ 1591.396839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_interrupt -r 2 [ 1591.425728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_interrupt -r 2 [ 1591.457870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.20@tcp 192.168.206.120@tcp [ 1591.482218] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1591.489804] LNet: 1 local NIs in recovery (showing 1): 192.168.206.20@tcp [ 1591.695513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1591.716918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1591.742172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1591.764813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1591.788588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1591.805801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1591.833019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1591.874181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1591.890749] LNet: 88045:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1591.912783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1591.944930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1591.970152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1591.996776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1592.026534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1592.048064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1592.491239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_dropped -r 2 [ 1592.516232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_dropped -r 2 [ 1592.548817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_dropped -r 2 [ 1592.585617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_dropped -r 2 [ 1592.614048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_dropped -r 2 [ 1592.655373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_dropped -r 2 [ 1592.690810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_dropped -r 2 [ 1592.749711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_dropped -r 2 [ 1592.788864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.20@tcp 192.168.206.120@tcp [ 1592.813836] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1592.970142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1592.992963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1593.021477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1593.048709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1593.075553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1593.097629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1593.135421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1593.164500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1593.186613] LNet: 88165:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1593.205823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1593.234310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1593.267674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1593.294725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1593.315104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1593.341294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1593.752764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_aborted -r 2 [ 1593.781819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_aborted -r 2 [ 1593.815851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_aborted -r 2 [ 1593.866906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_aborted -r 2 [ 1593.913690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_aborted -r 2 [ 1593.956275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_aborted -r 2 [ 1593.997833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_aborted -r 2 [ 1594.024265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_aborted -r 2 [ 1594.055855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.20@tcp 192.168.206.120@tcp [ 1594.094992] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1594.313127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1594.335216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1594.358674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1594.381226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1594.402989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1594.429264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1594.454393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1594.486776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1594.509151] LNet: 88285:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1594.525649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1594.547777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1594.568550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1594.595155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1594.619160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1594.648806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1595.100414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_no_route -r 2 [ 1595.136182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_no_route -r 2 [ 1595.170457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_no_route -r 2 [ 1595.198424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_no_route -r 2 [ 1595.234563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_no_route -r 2 [ 1595.269541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_no_route -r 2 [ 1595.324196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_no_route -r 2 [ 1595.349698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_no_route -r 2 [ 1595.376920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.20@tcp 192.168.206.120@tcp [ 1595.553899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1595.582213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1595.602222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1595.638734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1595.675762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1595.707637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1595.730717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1595.759247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1595.795482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1595.830311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1595.863929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1595.891986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1595.926903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1595.965586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1596.361052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e local_timeout -r 2 [ 1596.403215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e local_timeout -r 2 [ 1596.438304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e local_timeout -r 2 [ 1596.473573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e local_timeout -r 2 [ 1596.506676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e local_timeout -r 2 [ 1596.538678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e local_timeout -r 2 [ 1596.572942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e local_timeout -r 2 [ 1596.601333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e local_timeout -r 2 [ 1596.629814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.20@tcp 192.168.206.120@tcp [ 1596.648165] LNet: There was an unexpected network error while writing to 192.168.206.120: rc = -22 [ 1596.652332] LNet: Skipped 1 previous similar message [ 1596.859513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1596.888251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1596.922619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1596.950496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1596.985771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1597.007531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1597.036493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1597.072909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1597.098291] LNet: 88526:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1597.107780] LNet: 88526:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1597.132445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1597.174209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1597.225209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1597.270920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1597.317875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1597.360941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1597.843855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp -m GET -e remote_dropped -r 2 [ 1597.897226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.120@tcp1 -m GET -e remote_dropped -r 2 [ 1597.948465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp -m GET -e remote_dropped -r 2 [ 1597.985694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp -d 192.168.206.20@tcp1 -m GET -e remote_dropped -r 2 [ 1598.021703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp -m GET -e remote_dropped -r 2 [ 1598.053909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.120@tcp1 -m GET -e remote_dropped -r 2 [ 1598.087829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp -m GET -e remote_dropped -r 2 [ 1598.133546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.20@tcp1 -d 192.168.206.20@tcp1 -m GET -e remote_dropped -r 2 [ 1598.170729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.20@tcp 192.168.206.120@tcp [ 1598.190626] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.120@tcp [ 1598.366382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1598.393736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1598.427283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1598.458714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1598.497335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1598.539205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1598.575787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1598.622217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1598.677942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1598.736449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1598.773791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1598.821322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1598.863690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1598.893338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1603.245432] LNet: 88720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.255086] LNetError: 88720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.265163] LNet: Removed LNI 192.168.206.20@tcp [ 1603.287414] LNet: Removed LNI 192.168.206.20@tcp1 [ 1603.669772] Key type .llcrypt unregistered [ 1603.672811] Key type ._llcrypt unregistered [ 1618.747325] Key type ._llcrypt registered [ 1618.752360] Key type .llcrypt registered [ 1620.371991] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:39:57 (1783967997) [ 1620.894805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1620.917552] alg: No test for adler32 (adler32-zlib) [ 1621.919553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.013239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.043944] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1623.170368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1623.202555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.20@tcp [ 1623.222061] LNetError: 89701:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.206.20@tcp to recovery [ 1623.310811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.20@tcp [ 1623.429765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1630.140923] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:40:07 (1783968007) [ 1630.280335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.319887] LNet: 90286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.330554] LNetError: 90286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1630.356471] LNet: Removed LNI 192.168.206.20@tcp [ 1630.397433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.476799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.494754] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1631.387553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1631.425488] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1632.702730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.20@tcp [ 1632.728438] LNetError: 90559:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.206.20@tcp to recovery [ 1633.503698] LNet: 1 local NIs in recovery (showing 1): 192.168.206.20@tcp [ 1633.515540] LNet: There was an unexpected network error while writing to 192.168.206.20: rc = -22 [ 1634.046542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.20@tcp [ 1641.982968] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:40:20 (1783968020) [ 1643.838790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg620-server [ 1645.153726] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:40:23 (1783968023) [ 1646.701438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg620-server [ 1648.419821] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:40:26 (1783968026) [ 1650.294616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg620-server [ 1651.884979] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:40:29 (1783968029) [ 1652.245551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.260536] LNet: 92189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.269491] LNetError: 92189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.283300] LNet: Removed LNI 192.168.206.20@tcp [ 1652.301606] LNet: Removed LNI 192.168.206.20@tcp1 [ 1652.319596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.372184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1652.392538] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1653.188160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1653.242661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1653.268840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1653.322663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1653.348375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1653.374767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1659.329261] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:40:37 (1783968037) [ 1659.467701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.503807] LNet: 92909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.517310] LNetError: 92909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.526709] LNetError: 92909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1659.556434] LNet: Removed LNI 192.168.206.20@tcp [ 1659.588982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.633728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1659.653777] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1666.775844] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:40:44 (1783968044) [ 1667.062602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.083623] LNet: 93618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.094912] LNetError: 93618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.125351] LNet: Removed LNI 192.168.206.20@tcp [ 1667.163983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.260732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1667.291225] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1674.445122] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:40:52 (1783968052) [ 1676.024925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1677.524779] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:40:55 (1783968055) [ 1678.024298] LNet: 94654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.035080] LNetError: 94654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.054755] LNet: Removed LNI 192.168.206.20@tcp [ 1678.557130] Key type .llcrypt unregistered [ 1678.562271] Key type ._llcrypt unregistered [ 1684.050723] Key type ._llcrypt registered [ 1684.057664] Key type .llcrypt registered [ 1684.399328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1684.454550] alg: No test for adler32 (adler32-zlib) [ 1685.356412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1685.383115] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1692.739505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1697.827774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.120@tcp [ 1698.104253] LNet: 95071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.115550] LNetError: 95071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.135453] LNet: Removed LNI 192.168.206.20@tcp [ 1698.535511] Key type .llcrypt unregistered [ 1698.539594] Key type ._llcrypt unregistered [ 1708.446835] Key type ._llcrypt registered [ 1708.448443] Key type .llcrypt registered [ 1709.651458] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1711.018563] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1712.787951] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:41:30 (1783968090) [ 1714.380529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1716.549943] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:41:34 (1783968094) [ 1718.271214] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg620-server [ 1720.374234] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:41:38 (1783968098) [ 1722.357076] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg620-server [ 1724.701575] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:41:41 (1783968101) [ 1726.413162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1726.626487] alg: No test for adler32 (adler32-zlib) [ 1727.741197] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 1727.903613] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1729.559184] Key type lgssc registered [ 1730.435508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1748.677617] Key type lgssc unregistered [ 1748.979033] LNet: 98245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.985221] LNetError: 98245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.002229] LNet: Removed LNI 192.168.206.20@tcp [ 1749.653842] Key type .llcrypt unregistered [ 1749.656268] Key type ._llcrypt unregistered [ 1762.897768] Key type ._llcrypt registered [ 1762.901862] Key type .llcrypt registered [ 1764.102765] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:42:21 (1783968141) [ 1764.513025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1764.526428] alg: No test for adler32 (adler32-zlib) [ 1765.467721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.521685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1765.547552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1765.595027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1765.637786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1772.662388] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:42:30 (1783968150) [ 1772.777721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.804665] LNet: 99653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.835252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.933265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1772.977400] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1773.801326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.839939] LNet: 99804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.847860] LNetError: 99804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.870232] LNet: Removed LNI 192.168.206.20@tcp [ 1773.885486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.932436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1773.957213] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1774.478946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.507744] LNetError: 99832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.519629] LNet: Removed LNI 192.168.206.20@tcp [ 1774.546234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.611768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1774.654850] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1780.285353] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:42:38 (1783968158) [ 1780.383712] LNet: 100414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.400573] LNet: 100414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1780.409404] LNetError: 100414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.438059] LNet: Removed LNI 192.168.206.20@tcp [ 1781.006815] Key type .llcrypt unregistered [ 1781.013794] Key type ._llcrypt unregistered [ 1785.474650] Key type ._llcrypt registered [ 1785.478275] Key type .llcrypt registered [ 1785.636390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1785.650206] alg: No test for adler32 (adler32-zlib) [ 1786.575747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.658216] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1786.660684] LNet: *** cfs_fail_loc=e003, val=100*** [ 1787.304454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1787.336819] LNet: *** cfs_fail_loc=e003, val=100*** [ 1787.342851] LNet: Skipped 2 previous similar messages [ 1787.350377] Ping reply received with mapping: [ 1787.352912] found metadata reply NI[192.168.206.20@tcp] [ 1787.359534] 14 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1787.365221] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1787.369777] 5a 5a 5a 5a 5a 5a 5a 5a [ 1787.524169] LNet: 100819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.532869] LNetError: 100819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.583359] LNet: Removed LNI 192.168.206.20@tcp [ 1789.040229] Key type .llcrypt unregistered [ 1789.043900] Key type ._llcrypt unregistered [ 1798.004249] Key type ._llcrypt registered [ 1798.008201] Key type .llcrypt registered [ 1799.154831] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:42:57 (1783968177) [ 1799.582612] Key type .llcrypt unregistered [ 1799.586903] Key type ._llcrypt unregistered [ 1804.443654] Key type ._llcrypt registered [ 1804.446244] Key type .llcrypt registered [ 1804.678362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1804.695976] alg: No test for adler32 (adler32-zlib) [ 1805.714273] LNetError: 101864:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1805.974458] Key type .llcrypt unregistered [ 1805.977112] Key type ._llcrypt unregistered [ 1809.993797] Key type ._llcrypt registered [ 1809.997412] Key type .llcrypt registered [ 1811.093407] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:43:08 (1783968188) [ 1811.658409] Key type .llcrypt unregistered [ 1811.665169] Key type ._llcrypt unregistered [ 1816.625073] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1816.627295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1819.052184] Key type ._llcrypt registered [ 1819.055200] Key type .llcrypt registered [ 1819.212487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1819.223884] alg: No test for adler32 (adler32-zlib) [ 1820.110468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.150650] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1827.667846] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:43:25 (1783968205) [ 1827.887777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.907332] LNet: 103645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.913634] LNetError: 103645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1827.939425] LNet: Removed LNI 10.1.2.3@tcp [ 1827.957500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.569414] Key type .llcrypt unregistered [ 1828.574682] Key type ._llcrypt unregistered [ 1834.339332] Key type ._llcrypt registered [ 1834.341774] Key type .llcrypt registered [ 1834.487885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1834.510969] alg: No test for adler32 (adler32-zlib) [ 1835.425685] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1835.472135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1835.600849] LNet: 103933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.611709] LNetError: 103933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.630468] LNet: Removed LNI 192.168.206.20@tcp [ 1836.281819] Key type .llcrypt unregistered [ 1836.284424] Key type ._llcrypt unregistered [ 1846.165967] Key type ._llcrypt registered [ 1846.168559] Key type .llcrypt registered [ 1847.402704] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:43:45 (1783968225) [ 1847.932365] Key type .llcrypt unregistered [ 1847.936100] Key type ._llcrypt unregistered [ 1853.173674] Key type ._llcrypt registered [ 1853.175906] Key type .llcrypt registered [ 1853.458568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1853.477975] alg: No test for adler32 (adler32-zlib) [ 1860.118724] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:43:58 (1783968238) [ 1860.223986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.273936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.362789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1860.389489] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1861.105927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1861.134851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1861.164777] LNet: Removed LNI 192.168.206.20@tcp [ 1861.393365] LNet: 105677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1861.680491] Key type .llcrypt unregistered [ 1861.683436] Key type ._llcrypt unregistered [ 1872.780475] Key type ._llcrypt registered [ 1872.781897] Key type .llcrypt registered [ 1873.829314] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:44:11 (1783968251) [ 1874.361887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1874.371744] alg: No test for adler32 (adler32-zlib) [ 1875.251363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.330351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1875.363436] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1876.144082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1876.217738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.20@tcp [ 1876.237516] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1876.245543] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1876.248799] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1876, 0, 0 [ 1876.253655] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1876.257197] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1876.260596] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1876.263550] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1876, 0, 0 [ 1876.267623] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1876.271422] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1876.274372] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1876.279821] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1876, 0, 1926 [ 1876.289709] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1876.292909] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1876.296261] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1876.302433] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1876, 0, 1926 [ 1876.310690] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1882.643353] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:44:20 (1783968260) [ 1883.030241] LNet: 107217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.038484] LNetError: 107217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.062340] LNet: Removed LNI 192.168.206.20@tcp [ 1883.616219] Key type .llcrypt unregistered [ 1883.618830] Key type ._llcrypt unregistered [ 1889.731315] Key type ._llcrypt registered [ 1889.735070] Key type .llcrypt registered [ 1889.970921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1889.983778] alg: No test for adler32 (adler32-zlib) [ 1890.856777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1890.886497] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1897.478759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 1905.409711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.469279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1905.504276] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 1906.667112] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1906.820280] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1906.823241] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1906.829203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1909.210725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1909.236969] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1919.770977] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:44:57 (1783968297) [ 1920.099690] LNet: 108723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.108649] LNetError: 108723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.119652] LNet: Removed LNI 192.168.206.20@tcp [ 1921.183370] LNet: Removed LNI 192.168.206.20@tcp1 [ 1921.191400] LNetError: 108723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1921.201147] LNetError: 108723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1921.221052] LNet: Removed LNI 10.1.2.3@tcp99 [ 1922.231294] Key type .llcrypt unregistered [ 1922.238787] Key type ._llcrypt unregistered [ 1928.685079] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1928.689669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1931.183324] Key type ._llcrypt registered [ 1931.186302] Key type .llcrypt registered [ 1931.346128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1931.361694] alg: No test for adler32 (adler32-zlib) [ 1932.292522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.348891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1932.376697] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1932.976948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1933.003353] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1934.823606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1934.866153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1934.903965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1934.927970] LNetError: 109468:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1934.962392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1935.007752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 1935.463088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1935.537173] LNet: 109488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.543868] LNetError: 109488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1935.549226] LNet: Removed LNI 10.1.2.3@tcp [ 1936.614363] LNet: Removed LNI 192.168.206.20@tcp [ 1937.223547] Key type .llcrypt unregistered [ 1937.225890] Key type ._llcrypt unregistered [ 1947.548661] Key type ._llcrypt registered [ 1947.555818] Key type .llcrypt registered [ 1948.649366] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:45:26 (1783968326) [ 1948.991024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1949.008314] alg: No test for adler32 (adler32-zlib) [ 1949.956172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.999774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1950.024069] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1950.579264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.20@tcp [ 1950.638137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg620-client@tcp [ 1955.826130] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:45:34 (1783968334) [ 1956.089573] LNet: 111023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.107243] LNetError: 111023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.120159] LNet: Removed LNI 192.168.206.20@tcp [ 1956.601986] Key type .llcrypt unregistered [ 1956.606918] Key type ._llcrypt unregistered [ 1961.297379] Key type ._llcrypt registered [ 1961.299286] Key type .llcrypt registered [ 1961.497402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1961.527891] alg: No test for adler32 (adler32-zlib) [ 1962.409268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1962.427711] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1962.451957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1967.703338] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:45:45 (1783968345) [ 1968.770320] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1970.116925] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:45:48 (1783968348) [ 1971.384344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1972.972782] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:45:50 (1783968350) [ 1973.127133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.154913] LNet: 112436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.162552] LNetError: 112436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.177790] LNet: Removed LNI 192.168.206.20@tcp [ 1973.200752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.234783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1973.274676] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1973.740721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1973.786564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1980.368161] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:45:57 (1783968357) [ 1980.452499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.482310] LNet: 113143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.488853] LNetError: 113143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.501559] LNet: Removed LNI 192.168.206.20@tcp [ 1980.520809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.610960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1980.676958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1981.417589] Key type .llcrypt unregistered [ 1981.428481] Key type ._llcrypt unregistered [ 1992.323908] Key type ._llcrypt registered [ 1992.330101] Key type .llcrypt registered [ 1993.378423] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:46:11 (1783968371) [ 1993.897696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.911061] alg: No test for adler32 (adler32-zlib) [ 1994.820247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.876640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1994.911036] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 1995.543820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1995.624056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2001.924856] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:46:19 (1783968379) [ 2002.231296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.251390] LNet: 114707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.258253] LNetError: 114707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.282623] LNet: Removed LNI 192.168.206.20@tcp [ 2002.315615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.368535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2002.396586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2007.672572] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:46:25 (1783968385) [ 2007.772588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.809565] LNet: 115282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.856913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.938597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2008.276345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2008.314299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2008.395768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2008.889541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2008.963583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2009.130520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2009.161092] LNet: 115320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.208569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.359518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2009.943245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2009.988230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2010.065200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2010.411463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2010.441501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2016.724799] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:46:34 (1783968394) [ 2016.872356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2016.901532] LNet: 115921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.917670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.956881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2016.986530] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2017.639697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2017.652640] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2018.429757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2018.488658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2018.657428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2018.673408] LNetError: 116200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.679216] LNet: Removed LNI 192.168.206.20@tcp [ 2019.684502] LNetError: 116200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.710758] LNet: Removed LNI 192.168.206.20@tcp2 [ 2019.736793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2019.789881] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2019.802990] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2019.810880] LNet: 116206:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2020.294874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2020.456745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2020.500330] LNet: Removed LNI 192.168.206.20@tcp [ 2020.510846] LNet: Removed LNI 192.168.206.20@tcp2 [ 2020.531051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.578957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2020.615429] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2021.296931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2021.324408] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2022.176697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2022.227889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2022.322296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2022.555954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2022.576479] LNet: 116528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.581356] LNet: 116528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2022.586970] LNetError: 116528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.592162] LNetError: 116528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2022.597573] LNet: Removed LNI 192.168.206.20@tcp [ 2023.657444] LNet: Removed LNI 192.168.206.20@tcp2 [ 2023.716861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.820776] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2023.828686] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2023.839390] LNet: 116534:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2024.435563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2024.675337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2024.701360] LNet: Removed LNI 192.168.206.20@tcp [ 2025.766790] LNet: Removed LNI 192.168.206.20@tcp2 [ 2025.786150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2025.858078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2025.894882] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2026.514728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2026.540654] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2027.220755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2027.271937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2027.496329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2027.522585] LNetError: 116851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.527629] LNetError: 116851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2027.535852] LNet: Removed LNI 192.168.206.20@tcp [ 2027.550809] LNet: Removed LNI 192.168.206.20@tcp2 [ 2027.568533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.644558] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2027.653475] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2027.656828] LNet: 116857:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2028.094272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2028.262024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2029.343135] LNet: Removed LNI 192.168.206.20@tcp [ 2029.351766] LNet: Removed LNI 192.168.206.20@tcp2 [ 2029.369694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.424547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2029.454892] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2030.020688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2030.040993] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2030.766910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2030.813576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2030.881397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2031.102539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.135565] LNet: 117179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.143823] LNet: 117179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2032.159160] LNet: Removed LNI 192.168.206.20@tcp [ 2033.191198] LNet: Removed LNI 192.168.206.20@tcp2 [ 2033.213968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.282340] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2033.288723] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2033.293474] LNet: 117185:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2033.656799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2033.861797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2033.881265] LNet: Removed LNI 192.168.206.20@tcp [ 2034.925135] LNet: Removed LNI 192.168.206.20@tcp2 [ 2034.954775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.012051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2035.044801] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2035.741154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2035.763230] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2036.636991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2036.689385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2036.771626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2037.070640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2037.091804] LNetError: 117519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.096637] LNetError: 117519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2038.111989] LNet: Removed LNI 192.168.206.20@tcp [ 2038.141058] LNet: Removed LNI 192.168.206.20@tcp2 [ 2038.191236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.271376] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2038.285203] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2038.291566] LNet: 117525:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2038.717992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2039.025286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2040.095322] LNet: Removed LNI 192.168.206.20@tcp [ 2040.109222] LNet: Removed LNI 192.168.206.20@tcp2 [ 2040.182165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.270552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2040.308444] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2040.945842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2040.969589] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2041.710283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2041.753981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2041.791237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2041.851113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2042.148270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2043.231136] LNet: Removed LNI 192.168.206.20@tcp [ 2044.262313] LNet: Removed LNI 192.168.206.20@tcp2 [ 2044.305560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2044.426143] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2044.445505] LNet: Added LNI 192.168.206.20@tcp2 [8/256/0/180] [ 2045.226931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2051.645189] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2053.035856] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:47:11 (1783968431) [ 2053.216810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.237439] LNet: 118645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.243384] LNet: 118645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2053.251131] LNetError: 118645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.258446] LNetError: 118645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2053.267305] LNet: Removed LNI 192.168.206.20@tcp [ 2054.313456] LNet: Removed LNI 192.168.206.20@tcp2 [ 2054.342266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.607485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2054.630691] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2055.159614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2055.176576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2055.854519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2055.871109] LNet: Removed LNI 10.1.2.3@tcp [ 2055.881909] LNet: Removed LNI 192.168.206.20@tcp [ 2055.900839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.948541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2055.976708] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2055.981675] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2056.292437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2056.304623] LNet: Removed LNI 192.168.206.20@tcp [ 2056.315741] LNet: Removed LNI 10.1.2.3@tcp [ 2061.874899] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:47:19 (1783968439) [ 2061.961621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2062.018088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.104608] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2062.215886] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2062.223859] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2062.230252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2064.624503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2064.655721] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2065.403499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2065.417971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2067.456798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.20@tcp [ 2067.515768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2067.583485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.20@tcp -m PUT -e local_timeout -r 1 [ 2067.663580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2067.741702] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2067.745477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2067.978092] LNetError: 119583:0:(lib-move.c:4771:lnet_parse()) 192.168.206.20@tcp, src 192.168.206.20@tcp: Dropping PUT (error -108 looking up sender) [ 2069.024618] LNet: Removed LNI 10.1.2.3@tcp [ 2070.059915] LNet: Removed LNI 192.168.206.20@tcp [ 2070.486590] Key type .llcrypt unregistered [ 2070.490948] Key type ._llcrypt unregistered [ 2081.426528] Key type ._llcrypt registered [ 2081.428428] Key type .llcrypt registered [ 2082.536342] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:47:40 (1783968460) [ 2082.963347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.975515] alg: No test for adler32 (adler32-zlib) [ 2083.998906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.735517] LNet: 120895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.617673] Key type .llcrypt unregistered [ 2085.619428] Key type ._llcrypt unregistered [ 2092.219459] Key type ._llcrypt registered [ 2092.223112] Key type .llcrypt registered [ 2092.550151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2092.576401] alg: No test for adler32 (adler32-zlib) [ 2093.559725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2093.605129] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2101.233394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 2101.508338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2102.198869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.120@tcp [ 2102.581016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2102.642263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2102.679350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2102.700764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2102.751143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2102.797478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2102.816330] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.120@tcp [ 2102.858932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2102.893166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2102.916923] LNet: 121345:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2102.938930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2102.976644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2103.014387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2103.075364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2103.120729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2103.169579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2103.870323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2104.076334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2104.109887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2104.145604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2104.167439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2104.189866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2104.212713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2104.237290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2104.263324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2104.277304] LNet: 121462:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2104.291419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2104.314253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2104.341882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2104.371621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2104.393864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2104.413153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2105.146244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.120@tcp [ 2105.390893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2105.440047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2105.463849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2105.488921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2105.521252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2105.581929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2105.651834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2105.693536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2105.722813] LNet: 121579:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2105.749157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2105.797142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2105.838774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2105.889409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2105.937252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2105.989113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2106.548726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2106.888169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2106.933554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2106.986208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2107.060599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2107.118376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2107.171200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2107.195894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2107.230881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2107.263947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2107.294932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2107.340311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2107.380927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2107.432994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2107.495894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2107.910407] LNet: 121743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2107.924622] LNetError: 121743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.943736] LNet: Removed LNI 192.168.206.20@tcp [ 2108.486550] Key type .llcrypt unregistered [ 2108.493245] Key type ._llcrypt unregistered [ 2124.493675] Key type ._llcrypt registered [ 2124.496561] Key type .llcrypt registered [ 2125.667258] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:48:23 (1783968503) [ 2126.026596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.039530] alg: No test for adler32 (adler32-zlib) [ 2127.006865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.398538] LNet: 122632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.028827] Key type .llcrypt unregistered [ 2128.031956] Key type ._llcrypt unregistered [ 2134.230899] Key type ._llcrypt registered [ 2134.233397] Key type .llcrypt registered [ 2134.494125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2134.525569] alg: No test for adler32 (adler32-zlib) [ 2135.529422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2135.568707] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2142.765666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.120@tcp [ 2151.587343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.651452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2151.686931] LNet: Added LNI 192.168.206.20@tcp1 [8/256/0/180] [ 2152.421951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2152.991490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.120@tcp [ 2153.187571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2153.212724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2153.242304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2153.281878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2153.310309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2153.336937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2153.362645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2153.385778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2153.409768] LNet: 123285:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2153.434214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2153.484818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2153.530885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2153.569436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2153.615396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2153.643790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2154.294079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2154.499381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2154.520544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2154.550642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2154.574167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2154.597151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2154.627542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2154.656451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2154.689736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2154.717921] LNet: 123402:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2154.740558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2154.781934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2154.831590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2154.871459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2154.906799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2154.934044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2155.474189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.120@tcp [ 2155.712752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2155.745977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2155.803472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2155.852547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2155.888939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2155.927194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2155.951034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2155.992840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2156.011793] LNet: 123519:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2156.042849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2156.063484] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.120@tcp [ 2156.081367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2156.107737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2156.141896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2156.164513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2156.200612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2156.766236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2156.960972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2156.985554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2157.008439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2157.033681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2157.073506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2157.119597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2157.146746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2157.179495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2157.209467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2157.258171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2157.292655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2157.326261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2157.360376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2157.408193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2162.465465] LNet: 123717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2162.474510] LNetError: 123717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.479135] LNet: Removed LNI 192.168.206.20@tcp [ 2162.489611] LNet: Removed LNI 192.168.206.20@tcp1 [ 2162.880937] Key type .llcrypt unregistered [ 2162.884457] Key type ._llcrypt unregistered [ 2177.534994] Key type ._llcrypt registered [ 2177.537138] Key type .llcrypt registered [ 2178.841663] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 14:49:16 (1783968556) [ 2180.025246] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2181.528710] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:49:19 (1783968559) [ 2187.986417] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:49:25 (1783968565) [ 2193.909327] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:49:31 (1783968571) [ 2200.271699] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:49:38 (1783968578) [ 2206.664316] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:49:44 (1783968584) [ 2213.237528] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:49:50 (1783968590) [ 2219.407656] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:49:57 (1783968597) [ 2226.156821] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:50:03 (1783968603) [ 2232.309740] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:50:10 (1783968610) [ 2237.375765] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:50:15 (1783968615) [ 2243.557100] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:50:21 (1783968621) [ 2249.911562] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:50:27 (1783968627) [ 2256.084122] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:50:34 (1783968634) [ 2261.866966] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:50:39 (1783968639) [ 2268.882486] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:50:46 (1783968646) [ 2274.790986] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:50:52 (1783968652) [ 2279.998845] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:50:58 (1783968658) [ 2285.576026] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:51:03 (1783968663) [ 2285.925773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2285.944706] alg: No test for adler32 (adler32-zlib) [ 2286.816848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.934404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2286.975396] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2287.829573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.848685] LNet: 134715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.854903] LNetError: 134715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.883174] LNet: Removed LNI 192.168.206.20@tcp [ 2287.902799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.974714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2288.009823] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2294.041747] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:51:12 (1783968672) [ 2300.060652] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:51:18 (1783968678) [ 2307.827450] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:51:25 (1783968685) [ 2316.359119] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:51:34 (1783968694) [ 2323.492663] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:51:41 (1783968701) [ 2331.079443] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:51:48 (1783968708) [ 2337.479773] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:51:55 (1783968715) [ 2342.993638] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:52:01 (1783968721) [ 2348.010426] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:52:06 (1783968726) [ 2353.543190] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:52:11 (1783968731) [ 2358.705923] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:52:16 (1783968736) [ 2363.922392] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:52:21 (1783968741) [ 2369.461945] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2129 sec ========== 14:52:27 (1783968747) [ 2369.796444] LNet: 142177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.802915] LNetError: 142177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.824626] LNet: Removed LNI 192.168.206.20@tcp [ 2370.484625] Key type .llcrypt unregistered [ 2370.491282] Key type ._llcrypt unregistered [ 2380.239752] Key type ._llcrypt registered [ 2380.254728] Key type .llcrypt registered [ 2380.533269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2380.542406] alg: No test for adler32 (adler32-zlib) [ 2381.647529] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 2381.931099] LNet: Added LNI 192.168.206.20@tcp [8/256/0/180] [ 2383.567108] Key type lgssc registered [ 2384.508862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2438.327577] Lustre: Mounted lustre-client [ 2442.846560] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2450.640712] Lustre: Unmounted lustre-client [ 2469.410600] Key type lgssc unregistered [ 2469.619588] LNet: 145102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.624123] LNetError: 145102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.641047] LNet: Removed LNI 192.168.206.20@tcp [ 2470.236148] Key type .llcrypt unregistered [ 2470.240948] Key type ._llcrypt unregistered