[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 438432544 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003354] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008333] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.012132] LSM: Security Framework initializing [ 0.013058] Yama: becoming mindful. [ 0.014055] SELinux: Initializing. [ 0.015064] *** VALIDATE selinux *** [ 0.023682] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027847] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030087] *** VALIDATE tmpfs *** [ 0.031401] *** VALIDATE proc *** [ 0.033127] *** VALIDATE cgroup *** [ 0.034000] *** VALIDATE cgroup2 *** [ 0.034051] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041806] debug: unmapping init [mem 0xffffffffaba59000-0xffffffffaba60fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044694] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049019] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052273] rcu: Hierarchical SRCU implementation. [ 0.054432] smp: Bringing up secondary CPUs ... [ 0.055643] x86: Booting SMP configuration: [ 0.056087] .... node #0, CPUs: #1 #2 #3 [ 0.072549] smp: Brought up 1 node, 4 CPUs [ 0.074022] smpboot: Max logical packages: 1 [ 0.075026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.121957] node 0 deferred pages initialised in 41ms [ 0.123124] devtmpfs: initialized [ 0.125247] x86/mm: Memory block size: 128MB [ 0.129182] gcov: version magic: 0x41383552 [ 0.134327] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136000] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138282] pinctrl core: initialized pinctrl subsystem [ 0.144913] [ 0.147285] ************************************************************* [ 0.154019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161162] ** ** [ 0.165108] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168016] ** ** [ 0.172016] ** This means that this kernel is built to expose internal ** [ 0.175028] ** IOMMU data structures, which may compromise security on ** [ 0.179019] ** your system. ** [ 0.181018] ** ** [ 0.184015] ** If you see this message and you are not debugging the ** [ 0.187016] ** kernel, report this immediately to your vendor! ** [ 0.190018] ** ** [ 0.192017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195066] ************************************************************* [ 0.198000] NET: Registered protocol family 16 [ 0.204183] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.210307] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.222094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.236025] cpuidle: using governor menu [ 0.251044] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.263000] PCI: Using configuration type 1 for base access [ 0.268154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.285605] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.286000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.291120] cryptd: max_cpu_qlen set to 1000 [ 0.294480] ACPI: Added _OSI(Module Device) [ 0.297021] ACPI: Added _OSI(Processor Device) [ 0.302021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.307021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.322039] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.334066] ACPI: Interpreter enabled [ 0.335090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.338019] ACPI: Using IOAPIC for interrupt routing [ 0.343481] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.349737] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.375133] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.380057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.385025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.395186] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.410419] acpiphp: Slot [2] registered [ 0.413455] acpiphp: Slot [5] registered [ 0.414321] acpiphp: Slot [6] registered [ 0.419864] acpiphp: Slot [3] registered [ 0.424240] acpiphp: Slot [4] registered [ 0.428612] acpiphp: Slot [7] registered [ 0.433477] acpiphp: Slot [8] registered [ 0.436164] acpiphp: Slot [9] registered [ 0.438398] acpiphp: Slot [10] registered [ 0.441318] acpiphp: Slot [11] registered [ 0.443159] acpiphp: Slot [12] registered [ 0.445314] acpiphp: Slot [13] registered [ 0.448188] acpiphp: Slot [14] registered [ 0.452149] acpiphp: Slot [15] registered [ 0.453279] acpiphp: Slot [16] registered [ 0.458201] acpiphp: Slot [17] registered [ 0.462181] acpiphp: Slot [18] registered [ 0.465147] acpiphp: Slot [19] registered [ 0.469196] acpiphp: Slot [20] registered [ 0.470167] acpiphp: Slot [21] registered [ 0.477206] acpiphp: Slot [22] registered [ 0.479115] acpiphp: Slot [23] registered [ 0.483162] acpiphp: Slot [24] registered [ 0.488141] acpiphp: Slot [25] registered [ 0.493159] acpiphp: Slot [26] registered [ 0.496142] acpiphp: Slot [27] registered [ 0.498000] acpiphp: Slot [28] registered [ 0.503669] acpiphp: Slot [29] registered [ 0.505143] acpiphp: Slot [30] registered [ 0.509136] acpiphp: Slot [31] registered [ 0.514026] PCI host bridge to bus 0000:00 [ 0.520385] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.528028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.533227] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.536028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.541026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.546041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.550057] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.563144] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.574039] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.597000] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.610191] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.613028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.618012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.626021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.642856] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.650524] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.655054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.661224] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 10742 usecs [ 0.666000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.680021] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.706025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.730857] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.752108] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.768018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.781021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.812023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.840735] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.852023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.860021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.875020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.892705] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.897023] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.900539] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.904140] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.905308] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.910073] iommu: Default domain type: Passthrough [ 0.911554] SCSI subsystem initialized [ 0.913269] ACPI: bus type USB registered [ 0.916156] usbcore: registered new interface driver usbfs [ 0.919273] usbcore: registered new interface driver hub [ 0.922198] usbcore: registered new device driver usb [ 0.925306] pps_core: LinuxPPS API ver. 1 registered [ 0.928020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.932272] PTP clock support registered [ 0.934575] EDAC MC: Ver: 3.0.0 [ 0.935203] PCI: Using ACPI for IRQ routing [ 0.937212] NetLabel: Initializing [ 0.939019] NetLabel: domain hash size = 128 [ 0.941026] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.945035] NetLabel: unlabeled traffic allowed by default [ 0.948004] vgaarb: loaded [ 0.950482] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.953018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.961369] clocksource: Switched to clocksource kvm-clock [ 1.212489] VFS: Disk quotas dquot_6.6.0 [ 1.218841] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.222901] *** VALIDATE ramfs *** [ 1.227254] *** VALIDATE hugetlbfs *** [ 1.229916] pnp: PnP ACPI init [ 1.233843] pnp: PnP ACPI: found 6 devices [ 1.273477] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.282432] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.290993] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.300569] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.304838] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.310711] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.315622] NET: Registered protocol family 2 [ 1.327131] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.354594] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.361162] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.386107] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.401415] TCP: Hash tables configured (established 65536 bind 65536) [ 1.414424] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.421238] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.428057] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.436083] NET: Registered protocol family 1 [ 1.440781] RPC: Registered named UNIX socket transport module. [ 1.449269] RPC: Registered udp transport module. [ 1.454794] RPC: Registered tcp transport module. [ 1.465494] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.476562] NET: Registered protocol family 44 [ 1.479642] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.486459] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.496662] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.507718] pci 0000:00:01.0: quirk_isa_dma_hangs+0x0/0x20 took 10799 usecs [ 1.512471] PCI: CLS 0 bytes, default 64 [ 1.514525] Unpacking initramfs... [ 4.864159] debug: unmapping init [mem 0xffff91053cc64000-0xffff91053ffcffff] [ 4.873863] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.877043] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.881438] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.681746] Initialise system trusted keyrings [ 5.683274] Key type blacklist registered [ 5.685609] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.703103] zbud: loaded [ 5.713612] *** VALIDATE nfs *** [ 5.718107] *** VALIDATE nfs4 *** [ 5.724785] pstore: using deflate compression [ 5.740838] Platform Keyring initialized [ 5.966344] NET: Registered protocol family 38 [ 5.969601] Key type asymmetric registered [ 5.972381] Asymmetric key parser 'x509' registered [ 5.977762] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.984202] io scheduler mq-deadline registered [ 5.986830] io scheduler kyber registered [ 5.989905] io scheduler bfq registered [ 5.993499] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.999106] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.011431] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.019390] ACPI: Power Button [PWRF] [ 6.032169] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.055709] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.084818] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.131146] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.175372] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.190126] Non-volatile memory driver v1.3 [ 6.195212] Linux agpgart interface v0.103 [ 6.280396] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 6.286286] vda: detected capacity change from 0 to 76775424 [ 6.320552] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.329145] vdb: detected capacity change from 0 to 1073741824 [ 6.342838] libphy: Fixed MDIO Bus: probed [ 6.360964] usbcore: registered new interface driver usbserial_generic [ 6.364418] usbserial: USB Serial support registered for generic [ 6.373516] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.381717] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.383559] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.390168] mousedev: PS/2 mouse device common for all mice [ 6.396695] rtc_cmos 00:05: RTC can wake from S4 [ 6.400303] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.410884] rtc_cmos 00:05: registered as rtc0 [ 6.416436] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.422107] intel_pstate: CPU model not supported [ 6.431577] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.432306] hid: raw HID events driver (C) Jiri Kosina [ 6.445220] usbcore: registered new interface driver usbhid [ 6.447336] usbhid: USB HID core driver [ 6.449809] drop_monitor: Initializing network drop monitor service [ 6.453362] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.459934] Initializing XFRM netlink socket [ 6.460366] NET: Registered protocol family 10 [ 6.465926] Segment Routing with IPv6 [ 6.485170] NET: Registered protocol family 17 [ 6.497896] mpls_gso: MPLS GSO support [ 6.509139] RAS: Correctable Errors collector initialized. [ 6.512658] AVX version of gcm_enc/dec engaged. [ 6.515379] AES CTR mode by8 optimization enabled [ 6.741274] sched_clock: Marking stable (6741177907, 0)->(8212622051, -1471444144) [ 6.758839] registered taskstats version 1 [ 6.767394] Loading compiled-in X.509 certificates [ 6.769744] zswap: loaded using pool lzo/zbud [ 6.804976] Key type big_key registered [ 6.834397] Key type encrypted registered [ 6.837652] ima: No TPM chip found, activating TPM-bypass! [ 6.839968] ima: Allocated hash algorithm: sha1 [ 6.841708] ima: No architecture policies found [ 6.843103] evm: Initialising EVM extended attributes: [ 6.845800] evm: security.selinux [ 6.846960] evm: security.ima [ 6.849156] evm: security.capability [ 6.851206] evm: HMAC attrs: 0x1 [ 6.857108] rtc_cmos 00:05: setting system clock to 2026-09-10 04:51:03 UTC (1789015863) [ 6.865678] debug: unmapping init [mem 0xffffffffaca03000-0xffffffffacbfffff] [ 6.871470] debug: unmapping init [mem 0xffffffffab782000-0xffffffffaba58fff] [ 6.882146] Write protecting the kernel read-only data: 28672k [ 6.887216] debug: unmapping init [mem 0xffffffffa9e03000-0xffffffffa9ffffff] [ 6.890347] debug: unmapping init [mem 0xffffffffaa714000-0xffffffffaa7fffff] [ 6.988868] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.008437] systemd[1]: Detected virtualization kvm. [ 7.015143] systemd[1]: Detected architecture x86-64. [ 7.023455] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.089444] systemd[1]: No hostname configured. [ 7.094084] systemd[1]: Set hostname to . [ 7.096473] random: systemd: uninitialized urandom read (16 bytes read) [ 7.100782] systemd[1]: Initializing machine ID from random generator. [ 7.212700] random: ln: uninitialized urandom read (6 bytes read) [ 7.417991] random: systemd: uninitialized urandom read (16 bytes read) [ 7.422788] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 7.433162] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.452528] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.599546] device-mapper: uevent: version 1.0.3 [ 8.605646] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.758525] virtio_net virtio0 ens2: renamed from eth0 [ 11.417188] scsi host0: ata_piix [ 11.464060] scsi host1: ata_piix [ 11.466551] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 11.475608] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 16.380092] random: crng init done [ 16.381984] random: 7 urandom warning(s) missed due to ratelimiting [ 16.726615] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.991102] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.723803] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.871247] SELinux: Disabled at runtime. [ 22.998853] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.012518] systemd[1]: Detected virtualization kvm. [ 23.015856] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.841268] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.845160] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.864419] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.884251] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.901187] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.929653] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.952533] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 25.611428] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.696128] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.747374] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.779622] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.623208] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.703400] EDAC sbridge: Ver: 1.1.2 [ 31.767235] Key type dns_resolver registered [ 32.486332] NFS: Registering the id_resolver key type [ 32.489953] Key type id_resolver registered [ 32.491458] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg654-client login: [ 49.583540] hrtimer: interrupt took 4727419 ns [ 92.072515] libcfs: loading out-of-tree module taints kernel. [ 92.282786] Key type ._llcrypt registered [ 92.292830] Key type .llcrypt registered [ 93.062449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 93.071479] alg: No test for adler32 (adler32-zlib) [ 94.527678] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 95.736557] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 97.488276] Key type lgssc registered [ 99.203051] Lustre: Echo OBD driver; http://www.lustre.org/ [ 227.553769] Lustre: Mounted lustre-client - version 2.17.58_39_gf941a14 [ 232.144670] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.856943] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing check_logdir /tmp/testlogs/ [ 250.082650] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing yml_node [ 253.412623] Lustre: lustre-OST0000-osc-ffff91058547c800: disconnect after 24s idle [ 254.206387] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 256.441517] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 258.806142] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 260.225735] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 10 00:55:15 EDT 2026 [ 275.101323] Lustre: Unmounted lustre-client [ 307.215289] Key type lgssc unregistered [ 307.575806] LNet: 8407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.590243] LNetError: 8407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 307.633289] LNet: Removed LNI 192.168.206.54@tcp [ 308.652195] Key type .llcrypt unregistered [ 308.653848] Key type ._llcrypt unregistered [ 337.372620] Key type ._llcrypt registered [ 337.375286] Key type .llcrypt registered [ 337.540252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 337.553391] alg: No test for adler32 (adler32-zlib) [ 338.433517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 338.496537] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 338.514860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 341.884652] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 342.121954] LNet: 9287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.131422] LNetError: 9287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 342.156432] LNet: Removed LNI 192.168.206.54@tcp [ 342.804488] Key type .llcrypt unregistered [ 342.808220] Key type ._llcrypt unregistered [ 347.959513] Key type ._llcrypt registered [ 347.960861] Key type .llcrypt registered [ 349.148950] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:56:44 (1789016204) [ 349.491892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 349.500427] alg: No test for adler32 (adler32-zlib) [ 350.363943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.479927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.513433] LNet: 9694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.537929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.631919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 350.674455] LNet: 9701:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 357.753646] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:56:53 (1789016213) [ 357.828270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.853735] LNet: 10274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.890650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.997901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 363.710456] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:56:59 (1789016219) [ 363.783997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.813933] LNet: 10859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.840975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.926514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 370.060696] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:57:05 (1789016225) [ 370.167100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.204276] LNet: 11445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.233250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.337889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 376.226278] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:57:11 (1789016231) [ 376.383907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.415988] LNet: 12035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.457201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.552354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 377.129700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 384.367177] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:57:19 (1789016239) [ 384.517424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.538730] LNet: 12621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.563724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.654210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 385.185475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 385.649635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 392.220877] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:57:27 (1789016247) [ 392.458597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.518500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.638824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 400.410845] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:57:35 (1789016255) [ 400.504738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.561956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.716244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 400.785774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 401.206299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 401.268400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 401.515112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 401.558643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 401.967253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 402.023726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 402.372382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 402.402333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 402.762940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 402.797395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 403.298974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 403.340395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 403.645397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 403.676704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 403.980192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 404.004580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 404.398081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 404.451907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 404.790567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 404.827979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 405.268718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 405.313738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 405.807902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 405.852074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 412.637933] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:57:47 (1789016267) [ 412.722602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.744357] LNet: 14484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.747934] LNet: 14484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 412.764670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.846994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 412.881437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 420.231362] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:57:55 (1789016275) [ 420.496313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.555732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.706211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 420.759051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 427.373724] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:58:02 (1789016282) [ 427.628837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.675713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.787354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 427.830710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 438.938494] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:58:13 (1789016293) [ 439.019388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.074382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.172839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 439.200884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 445.122331] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:58:20 (1789016300) [ 445.245508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.277208] LNet: 16827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.284080] LNet: 16827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 445.319973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.448168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 445.474222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 452.435446] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:58:27 (1789016307) [ 452.565680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.643884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.700665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 452.737954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 459.809436] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:58:34 (1789016314) [ 459.925675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.977799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.103656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.577984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.625982] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:58:43 (1789016323) [ 468.839712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.966355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.117551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.243315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.148334] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:58:51 (1789016331) [ 476.502578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.542859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.669612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.817161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.692348] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:58:58 (1789016338) [ 482.823871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.922175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.414175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.766622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.448962] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:59:05 (1789016345) [ 490.685777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.831387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.939972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.000708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.255423] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:59:12 (1789016352) [ 497.564210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.640267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.700234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.769461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.057302] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:59:20 (1789016360) [ 505.323892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.359483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.479536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 511.856421] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:59:27 (1789016367) [ 512.342280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.367847] LNet: 22274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.384145] LNet: 22274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 512.426610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.540724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 518.883706] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:59:34 (1789016374) [ 519.144162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.176358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.249361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 519.760499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 527.404708] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:59:42 (1789016382) [ 527.529372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.617354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.819564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 527.849287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 536.098308] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:59:51 (1789016391) [ 536.510985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.574961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.664519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 536.696308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 544.097619] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:59:59 (1789016399) [ 544.393633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.443984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.523173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 544.556679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 550.374132] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:00:05 (1789016405) [ 550.487767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.536379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.615769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 550.644206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 555.876983] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:00:11 (1789016411) [ 556.122812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.174744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.242286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 556.328848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 556.345520] LNetError: 25807:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 556.399966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 556.434978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 562.687857] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:00:17 (1789016417) [ 562.942197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.967270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.027056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Ik92XFNsox [ 563.092121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Ik92XFNsox [ 563.164712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Ik92XFNsox [ 570.161590] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:00:25 (1789016425) [ 570.249288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.292344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.383282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 570.421958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 570.472994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 577.506491] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:00:32 (1789016432) [ 577.976049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.044820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.148269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 578.182960] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 579.800544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 581.237840] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 583.336499] Key type lgssc registered [ 584.592531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 588.050944] LNetError: 27568:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.206.54@tcp99 has no available nets [ 590.215659] Key type lgssc unregistered [ 590.632499] LNetError: 28674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 590.653254] LNet: Removed LNI 192.168.206.54@tcp [ 591.386306] Key type .llcrypt unregistered [ 591.388558] Key type ._llcrypt unregistered [ 597.110450] Key type ._llcrypt registered [ 597.116191] Key type .llcrypt registered [ 598.275274] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:00:53 (1789016453) [ 599.279255] Key type .llcrypt unregistered [ 599.281274] Key type ._llcrypt unregistered [ 606.671804] Key type ._llcrypt registered [ 606.673901] Key type .llcrypt registered [ 606.888321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 606.922923] alg: No test for adler32 (adler32-zlib) [ 607.894753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.965737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 607.986803] LNet: 29596:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 607.997994] LNet: 29596:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 608.002755] LNet: 29596:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 608.024992] LNet: Message forwarding will be enabled in 2s [ 611.104715] LNet: Message forwarding enabled [ 611.565578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 611.594933] LNet: Message forwarding will stop in 2s to 6s [ 615.211493] LNet: Message forwarding disabled [ 616.122773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 616.144242] LNet: 29695:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 616.151910] LNet: 29695:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 616.157310] LNet: 29695:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 616.172273] LNet: Message forwarding will be enabled in 2s [ 616.329989] LNet: 29708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.944820] Key type .llcrypt unregistered [ 616.947866] Key type ._llcrypt unregistered [ 627.879757] Key type ._llcrypt registered [ 627.882324] Key type .llcrypt registered [ 629.104634] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:01:24 (1789016484) [ 629.518428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 629.570691] alg: No test for adler32 (adler32-zlib) [ 630.541403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.624641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 636.141790] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:01:31 (1789016491) [ 636.321862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.351543] LNet: 31108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.371639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.427491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 636.466330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 636.514545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 636.548859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 636.585357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 636.620301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 636.657504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 636.700328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 636.738573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 636.773222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 636.797149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 636.818859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 636.843587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 636.872776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 636.903525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 636.939608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 636.963565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 637.017669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 637.062721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 637.101529] LNetError: 31151:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 642.508666] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:01:37 (1789016497) [ 642.843285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.861696] LNet: 31722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.884283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.948294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.323826] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:01:44 (1789016504) [ 651.261509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg654-server [ 653.078837] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:01:48 (1789016508) [ 653.220937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.260692] LNet: 32650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.313758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.367220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 653.394141] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 654.337989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[55-59]@tcp [ 654.390538] LNet: 32785:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 654.448446] LNetError: 32656:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.55@tcp is being used as a gateway but routing feature is not turned on [ 654.479381] LNetError: 32656:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.55@tcp has gone from up to down [ 660.435520] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:01:56 (1789016516) [ 661.974926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg654-server [ 663.550520] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:01:58 (1789016518) [ 663.645777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.677473] LNet: 33717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.697453] LNetError: 33717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.746017] LNet: Removed LNI 192.168.206.54@tcp [ 663.783410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.865396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 663.884054] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 664.624882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[55-60/2]@tcp [ 664.666847] LNet: 33840:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 664.683364] LNet: 33840:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 664.701435] LNetError: 33727:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.55@tcp is being used as a gateway but routing feature is not turned on [ 664.719044] LNetError: 33727:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 664.732315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[55-60/2]@tcp [ 664.744309] LNetError: 33727:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.55@tcp has gone from up to down [ 664.744323] LNetError: 33727:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 670.281456] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:02:05 (1789016525) [ 670.677610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.724628] LNet: 34433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.747420] LNetError: 34433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 670.773173] LNet: Removed LNI 192.168.206.54@tcp [ 670.800245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.900393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 670.985130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.048974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.131518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.206181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.249717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.304240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 671.426632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.497775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.570515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.672363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.707305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.743332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 671.859190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.904301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.987613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.062685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.114317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.194202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 672.308743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.365505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.518995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.659646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.760750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.829060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 672.906464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.957685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.026937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.116457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.158715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.216778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 673.296092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.319573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.365535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 680.583752] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:02:15 (1789016535) [ 683.108920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg654-server [ 684.977738] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:02:20 (1789016540) [ 687.338325] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg654-server [ 689.853742] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:02:24 (1789016544) [ 689.998078] LNet: 35827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.010766] LNet: 35827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 690.928907] Key type .llcrypt unregistered [ 690.937667] Key type ._llcrypt unregistered [ 695.626468] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 695.690099] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 695.695469] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 695.699739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 698.465691] Key type ._llcrypt registered [ 698.471151] Key type .llcrypt registered [ 698.655982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 698.672634] alg: No test for adler32 (adler32-zlib) [ 699.544041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.592091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 699.607527] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 700.198129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 700.235520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 702.257315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 702.277173] LNet: Removed LNI 10.1.2.3@tcp [ 702.403846] LNet: 36553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.419304] LNetError: 36553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.447091] LNet: Removed LNI 192.168.206.54@tcp [ 702.912541] Key type .llcrypt unregistered [ 702.916454] Key type ._llcrypt unregistered [ 713.340205] Key type ._llcrypt registered [ 713.345265] Key type .llcrypt registered [ 714.645088] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:02:49 (1789016569) [ 715.501530] Key type .llcrypt unregistered [ 715.504514] Key type ._llcrypt unregistered [ 721.853233] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 722.126336] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 722.129951] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 722.137853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 724.767269] Key type ._llcrypt registered [ 724.769909] Key type .llcrypt registered [ 724.982560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 725.000466] alg: No test for adler32 (adler32-zlib) [ 725.891801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.968929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.991784] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 726.791550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 726.821771] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.330760] LNet: 38037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.334896] LNetError: 38037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 729.339470] LNet: Removed LNI 10.1.2.3@tcp [ 729.357370] LNet: Removed LNI 192.168.206.54@tcp [ 729.882472] Key type .llcrypt unregistered [ 729.888538] Key type ._llcrypt unregistered [ 742.100928] Key type ._llcrypt registered [ 742.103022] Key type .llcrypt registered [ 743.362960] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:03:18 (1789016598) [ 743.996920] Key type .llcrypt unregistered [ 743.999357] Key type ._llcrypt unregistered [ 750.136327] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 750.287522] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 750.290090] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 750.293627] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 752.947706] Key type ._llcrypt registered [ 752.951820] Key type .llcrypt registered [ 753.207625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.231185] alg: No test for adler32 (adler32-zlib) [ 754.172863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.234735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 754.277257] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 756.231687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.261091] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 758.536105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 758.553414] LNet: Removed LNI 10.1.2.3@tcp [ 758.565613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 758.594960] LNet: Removed LNI 10.1.2.31@tcp [ 758.616928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 758.680933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.906948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 759.960204] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 761.256819] LNet: 39946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.269790] LNetError: 39946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 761.278930] LNet: Removed LNI 10.1.2.31@tcp [ 761.292233] LNet: Removed LNI 10.1.2.3@tcp [ 761.977721] Key type .llcrypt unregistered [ 761.985287] Key type ._llcrypt unregistered [ 775.527460] Key type ._llcrypt registered [ 775.529262] Key type .llcrypt registered [ 777.152664] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:03:51 (1789016631) [ 777.560902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.577712] alg: No test for adler32 (adler32-zlib) [ 778.506485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.565364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.587299] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 779.407777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.439855] LNet: 40897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.445790] LNetError: 40897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.469988] LNet: Removed LNI 192.168.206.54@tcp [ 779.494435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.563689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.54@tcp [ 779.581704] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 780.353798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.54@tcp [ 780.397223] LNet: Removed LNI 192.168.206.54@tcp [ 780.494858] LNet: 41033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.146886] Key type .llcrypt unregistered [ 781.149610] Key type ._llcrypt unregistered [ 792.137761] Key type ._llcrypt registered [ 792.140707] Key type .llcrypt registered [ 793.434491] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:04:08 (1789016648) [ 795.339049] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg654-server [ 796.685626] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:04:12 (1789016652) [ 797.207448] Key type .llcrypt unregistered [ 797.215881] Key type ._llcrypt unregistered [ 802.818862] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.133431] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.137179] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.141625] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.626620] Key type ._llcrypt registered [ 805.632165] Key type .llcrypt registered [ 805.777353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 805.785927] alg: No test for adler32 (adler32-zlib) [ 806.640364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.684266] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 807.406250] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 809.301813] LNet: 42864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.308505] LNetError: 42864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.312777] LNet: Removed LNI 192.168.206.54@tcp1 [ 809.325970] LNet: Removed LNI 10.1.2.3@tcp2 [ 809.872404] Key type .llcrypt unregistered [ 809.874100] Key type ._llcrypt unregistered [ 821.917451] Key type ._llcrypt registered [ 821.919730] Key type .llcrypt registered [ 823.244264] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:04:38 (1789016678) [ 824.197649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 824.215488] alg: No test for adler32 (adler32-zlib) [ 825.158430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.217813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 825.251707] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 826.286576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.304701] LNet: 43833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.314363] LNetError: 43833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.331643] LNet: Removed LNI 192.168.206.54@tcp [ 826.357660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.476493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 826.518869] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 834.513134] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:04:49 (1789016689) [ 834.947899] LNet: 44425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.972687] LNetError: 44425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.995629] LNet: Removed LNI 192.168.206.54@tcp [ 835.781589] Key type .llcrypt unregistered [ 835.784271] Key type ._llcrypt unregistered [ 842.647222] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 842.932814] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 842.937711] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 842.942674] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.504757] Key type ._llcrypt registered [ 845.506987] Key type .llcrypt registered [ 845.633740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 845.645383] alg: No test for adler32 (adler32-zlib) [ 846.616283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.689433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 846.724757] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 847.489863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 847.512873] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 849.231671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.247989] LNet: 45165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.262577] LNetError: 45165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 849.269752] LNet: Removed LNI 10.1.2.3@tcp [ 849.291256] LNet: Removed LNI 192.168.206.54@tcp [ 849.311657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.391902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 849.413044] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 849.417399] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 849.829257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.866480] LNet: 45192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.873286] LNetError: 45192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 849.878760] LNetError: 45192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 849.885748] LNet: Removed LNI 10.1.2.3@tcp [ 849.896992] LNet: Removed LNI 192.168.206.54@tcp [ 849.920857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.000807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 850.033184] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 850.037595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 850.443335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 850.462313] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 851.399750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.418958] LNet: 45354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.426827] LNetError: 45354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 851.438318] LNetError: 45354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 851.452909] LNet: Removed LNI 10.1.2.3@tcp [ 851.461805] LNet: Removed LNI 192.168.206.54@tcp [ 851.486605] LNet: Removed LNI 192.168.206.54@tcp2 [ 851.515726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.606914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 851.635632] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 851.646188] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 851.651058] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 858.883582] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:05:14 (1789016714) [ 859.172533] LNet: 45952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.181497] LNetError: 45952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 859.193742] LNetError: 45952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 859.202535] LNet: Removed LNI 10.1.2.3@tcp [ 859.209849] LNet: Removed LNI 192.168.206.54@tcp [ 859.228514] LNet: Removed LNI 192.168.206.54@tcp2 [ 859.873677] Key type .llcrypt unregistered [ 859.875904] Key type ._llcrypt unregistered [ 865.096428] Key type ._llcrypt registered [ 865.098773] Key type .llcrypt registered [ 865.304505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 865.319771] alg: No test for adler32 (adler32-zlib) [ 866.329317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.438933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.471121] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 867.328308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.348469] LNet: 46344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.354660] LNetError: 46344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.388161] LNet: Removed LNI 192.168.206.54@tcp [ 867.409691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.495900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 867.521568] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 868.173597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.237344] LNet: 46374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.267139] LNetError: 46374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.301158] LNet: Removed LNI 192.168.206.54@tcp [ 868.343488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.444691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 868.529914] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 869.241898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.270664] LNet: 46404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.286499] LNetError: 46404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.323930] LNet: Removed LNI 192.168.206.54@tcp [ 869.355976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.489237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 869.539596] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 870.327323] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 870.578601] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 870.594830] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 870.605473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 871.966539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.997761] LNet: 46476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.012786] LNetError: 46476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.048892] LNet: Removed LNI 192.168.206.54@tcp [ 872.089685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.193683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 872.235953] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 880.336520] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:05:35 (1789016735) [ 880.569991] LNet: 47070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.584338] LNetError: 47070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.616068] LNet: Removed LNI 192.168.206.54@tcp [ 881.363285] Key type .llcrypt unregistered [ 881.365063] Key type ._llcrypt unregistered [ 887.218451] Key type ._llcrypt registered [ 887.226221] Key type .llcrypt registered [ 887.416430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 887.428216] alg: No test for adler32 (adler32-zlib) [ 888.312201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.388385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.441289] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 889.314474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.332266] LNet: 47471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.338360] LNetError: 47471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.352533] LNet: Removed LNI 192.168.206.54@tcp [ 889.373914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.425812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 889.452619] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 895.375141] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:05:50 (1789016750) [ 895.503743] LNet: 48058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.510368] LNetError: 48058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.533601] LNet: Removed LNI 192.168.206.54@tcp [ 896.131855] Key type .llcrypt unregistered [ 896.136310] Key type ._llcrypt unregistered [ 901.098227] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 901.283847] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 901.288716] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 901.294650] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 903.967843] Key type ._llcrypt registered [ 903.970465] Key type .llcrypt registered [ 904.172197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.191085] alg: No test for adler32 (adler32-zlib) [ 905.076100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.149349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 905.184883] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 905.937746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.958486] LNet: 48512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.964876] LNetError: 48512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.989423] LNet: Removed LNI 192.168.206.54@tcp [ 906.016806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.088873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 906.121899] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 911.856085] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:06:07 (1789016767) [ 911.944885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.990129] LNet: 49104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.006727] LNetError: 49104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.037081] LNet: Removed LNI 192.168.206.54@tcp [ 912.063240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.172156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 917.918669] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:06:13 (1789016773) [ 918.128752] LNet: 49692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.668430] Key type .llcrypt unregistered [ 918.670562] Key type ._llcrypt unregistered [ 924.142445] Key type ._llcrypt registered [ 924.145192] Key type .llcrypt registered [ 924.358251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 924.369648] alg: No test for adler32 (adler32-zlib) [ 925.302446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.418673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.456346] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 926.483582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.501317] LNet: 50086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.508643] LNetError: 50086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.531295] LNet: Removed LNI 192.168.206.54@tcp [ 926.558371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.637370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 926.660701] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 932.682914] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:06:27 (1789016787) [ 932.766614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.784445] LNet: 50673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.788921] LNetError: 50673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.800912] LNet: Removed LNI 192.168.206.54@tcp [ 932.827211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.896188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 938.438936] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:06:33 (1789016793) [ 939.698810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 941.023360] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:06:36 (1789016796) [ 942.359117] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 943.761719] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:06:39 (1789016799) [ 945.140720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 946.567406] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:06:42 (1789016802) [ 947.806957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 949.360545] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:06:44 (1789016804) [ 950.364314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 951.798934] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:06:47 (1789016807) [ 952.010237] LNet: 52698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.569499] Key type .llcrypt unregistered [ 952.575459] Key type ._llcrypt unregistered [ 958.389036] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.639758] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 958.648726] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 958.653088] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.322827] Key type ._llcrypt registered [ 961.325569] Key type .llcrypt registered [ 961.597981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.627613] alg: No test for adler32 (adler32-zlib) [ 962.570775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.644833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.674485] LNet: Added LNI 192.168.206.54@tcp [8/512/0/180] [ 962.694622] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 962.703144] LNet: 53018:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 962.806492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.843929] LNet: 53037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.855577] LNetError: 53037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 962.880164] LNet: Removed LNI 10.1.2.3@tcp [ 962.881679] LNetError: 53037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.901725] LNet: Removed LNI 192.168.206.54@tcp [ 962.949106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.038324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 963.056944] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 963.131681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.153243] LNet: 53061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.163607] LNetError: 53061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.179669] LNet: Removed LNI 192.168.206.54@tcp [ 963.197185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.249405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 963.267693] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 963.272215] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 963.275469] LNet: 53069:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 963.322877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.334130] LNet: 53087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.337764] LNetError: 53087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 963.341743] LNet: Removed LNI 10.1.2.3@tcp [ 963.344515] LNetError: 53087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.354132] LNet: Removed LNI 192.168.206.54@tcp [ 963.372552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.469552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 963.512753] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 963.538724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 963.547264] LNet: 53095:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 963.861421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.898898] LNet: 53121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.915058] LNetError: 53121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 963.930405] LNet: Removed LNI 10.1.2.3@tcp [ 963.932282] LNetError: 53121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.974420] LNet: Removed LNI 192.168.206.54@tcp [ 964.027278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.099595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 964.140110] LNet: Added LNI 192.168.206.54@tcp [8/256/0/182] [ 964.151375] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 964.365525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.390283] LNet: 53149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.404772] LNetError: 53149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 964.411048] LNet: Removed LNI 10.1.2.3@tcp [ 964.414219] LNetError: 53149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.439692] LNet: Removed LNI 192.168.206.54@tcp [ 964.461685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.519718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 964.553064] LNet: Added LNI 192.168.206.54@tcp [8/256/0/182] [ 964.562573] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 964.569754] LNet: 53157:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 964.639175] LNet: Removed LNI 10.1.2.3@tcp [ 964.654888] LNet: Removed LNI 192.168.206.54@tcp [ 965.392796] Key type .llcrypt unregistered [ 965.397978] Key type ._llcrypt unregistered [ 975.191309] Key type ._llcrypt registered [ 975.192813] Key type .llcrypt registered [ 976.619814] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:07:11 (1789016831) [ 977.856559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 979.396918] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:07:14 (1789016834) [ 979.791316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 979.815392] alg: No test for adler32 (adler32-zlib) [ 980.713211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.752079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 980.777717] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 981.350211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.54 [ 981.368283] LNet: 54418:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 981.381832] LNet: Removed LNI 192.168.206.54@tcp [ 986.676447] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:07:22 (1789016842) [ 986.783186] LNet: 54982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.791702] LNetError: 54982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.809519] LNet: Removed LNI 192.168.206.54@tcp [ 987.386537] Key type .llcrypt unregistered [ 987.390511] Key type ._llcrypt unregistered [ 992.022796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 992.274558] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 994.835811] Key type ._llcrypt registered [ 994.841577] Key type .llcrypt registered [ 995.006409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 995.023124] alg: No test for adler32 (adler32-zlib) [ 996.210795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1001.204984] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:07:36 (1789016856) [ 1001.389320] LNet: 55915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.395617] LNetError: 55915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1001.414789] LNet: Removed LNI 10.1.2.3@tcp [ 1001.999622] Key type .llcrypt unregistered [ 1002.002825] Key type ._llcrypt unregistered [ 1006.309960] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1006.524778] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1006.527651] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1006.533711] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1009.079961] Key type ._llcrypt registered [ 1009.081951] Key type .llcrypt registered [ 1009.236477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.249172] alg: No test for adler32 (adler32-zlib) [ 1010.138466] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1015.137285] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:07:50 (1789016870) [ 1016.823886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1018.188733] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:07:53 (1789016873) [ 1018.388970] LNet: 57097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.397870] LNetError: 57097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1018.413514] LNet: Removed LNI 10.1.2.3@tcp [ 1018.894473] Key type .llcrypt unregistered [ 1018.897509] Key type ._llcrypt unregistered [ 1023.850764] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1024.080716] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1024.084112] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1024.088699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1026.749833] Key type ._llcrypt registered [ 1026.752123] Key type .llcrypt registered [ 1026.924133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1026.941522] alg: No test for adler32 (adler32-zlib) [ 1027.906789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.977265] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1035.540757] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:08:10 (1789016890) [ 1036.124235] LNet: 58323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.132651] LNetError: 58323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1036.152835] LNet: Removed LNI 10.1.2.3@tcp [ 1036.840128] Key type .llcrypt unregistered [ 1036.842550] Key type ._llcrypt unregistered [ 1042.346086] Key type ._llcrypt registered [ 1042.349449] Key type .llcrypt registered [ 1042.529187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1042.560466] alg: No test for adler32 (adler32-zlib) [ 1043.589262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1043.640387] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1051.414722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1052.004306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_interrupt -r 1 [ 1052.061994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_interrupt -r 1 [ 1052.118364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1052.141196] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1052.329856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.353384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.377700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.408633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.435683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1052.485523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.516419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.549594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.584252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.603702] LNet: 58760:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1052.622757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.653635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.682509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.713628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.754362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.789381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.829733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1053.324371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_dropped -r 1 [ 1053.356830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_dropped -r 1 [ 1053.396954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1053.417594] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1053.631936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.678552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1053.707601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.743373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.793436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1053.836860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.866890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.908087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.921932] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1053.956911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.971336] LNet: 58871:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.986981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.013947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.050388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.078499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.110088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.142419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.175691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1054.630226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_aborted -r 1 [ 1054.686688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_aborted -r 1 [ 1054.726433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1054.747921] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1054.956795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.984788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1055.007834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.031708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.057478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1055.082407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.105677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.130812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.155637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.169097] LNet: 58982:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1055.185610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1055.219198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1055.245960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1055.273841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1055.296805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1055.321802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1055.345573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1055.743254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_no_route -r 1 [ 1055.767527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_no_route -r 1 [ 1055.788669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1055.966758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1056.004823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1056.030421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1056.059464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1056.082208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1056.111502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1056.134158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1056.165199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1056.195673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1056.222890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1056.252975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.278589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.309038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.333649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.369266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.388816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1056.777222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_timeout -r 1 [ 1056.818650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_timeout -r 1 [ 1056.854706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1056.873584] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1056.878894] LNet: Skipped 2 previous similar messages [ 1057.063270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.095497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1057.134516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.167428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.199847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1057.238724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.266516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.290122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.318698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.333801] LNet: 59205:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1057.342095] LNet: 59205:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1057.359529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1057.386099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1057.426283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1057.459026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1057.503048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.529828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1057.573334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1057.955881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_error -r 1 [ 1057.991361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_error -r 1 [ 1058.026688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1058.200155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1058.225499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1058.255171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1058.294892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1058.328379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1058.361945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1058.410943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1058.452206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1058.491074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1058.535570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1058.577852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.615231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.652230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.701521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.723526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.746507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1058.963612] LNet: 59357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.972313] LNetError: 59357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.985567] LNet: Removed LNI 192.168.206.54@tcp [ 1059.271460] Key type .llcrypt unregistered [ 1059.275026] Key type ._llcrypt unregistered [ 1070.491152] Key type ._llcrypt registered [ 1070.492861] Key type .llcrypt registered [ 1071.447420] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:08:47 (1789016927) [ 1072.092791] Key type .llcrypt unregistered [ 1072.097759] Key type ._llcrypt unregistered [ 1076.882910] Key type ._llcrypt registered [ 1076.885494] Key type .llcrypt registered [ 1077.016294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1077.030774] alg: No test for adler32 (adler32-zlib) [ 1077.930878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1077.973203] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1083.570158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1092.662719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.700431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1092.727783] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1093.774556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_interrupt -r 1 [ 1093.813680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_interrupt -r 1 [ 1093.841612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_interrupt -r 1 [ 1093.869162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_interrupt -r 1 [ 1093.895166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_interrupt -r 1 [ 1093.919188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_interrupt -r 1 [ 1093.953312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_interrupt -r 1 [ 1093.998985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_interrupt -r 1 [ 1094.040871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1094.069109] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1094.080172] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1094.283751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1094.315310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1094.358499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.395137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1094.422073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1094.455823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1094.492551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1094.520793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1094.550326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1094.566940] LNet: 60820:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1094.581356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1094.615752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1094.651967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1094.679953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1094.703767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1094.733463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1094.772434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1095.208473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_dropped -r 1 [ 1095.242818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_dropped -r 1 [ 1095.281842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_dropped -r 1 [ 1095.310148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_dropped -r 1 [ 1095.345140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_dropped -r 1 [ 1095.381794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_dropped -r 1 [ 1095.433555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_dropped -r 1 [ 1095.464487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_dropped -r 1 [ 1095.517278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1095.535759] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1095.541198] LNet: Skipped 2 previous similar messages [ 1095.720592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1095.754831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1095.784893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1095.819832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1095.867672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1095.904640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1095.939047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1095.980994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1096.019846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1096.040602] LNet: 60943:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1096.066195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1096.115684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1096.157404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1096.183974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1096.212241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1096.232665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1096.259081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1096.670336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_aborted -r 1 [ 1096.720570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_aborted -r 1 [ 1096.777772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_aborted -r 1 [ 1096.830721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_aborted -r 1 [ 1096.877564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_aborted -r 1 [ 1096.916397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_aborted -r 1 [ 1096.960825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_aborted -r 1 [ 1097.022580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_aborted -r 1 [ 1097.064737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1097.090164] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1097.095144] LNet: Skipped 2 previous similar messages [ 1097.284845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.317345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1097.364475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.406904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.447217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1097.483456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.507886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.536730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.568281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.581510] LNet: 61067:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1097.599757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.619390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.638325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.663933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.693601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.716711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.740566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1098.241375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_no_route -r 1 [ 1098.268977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_no_route -r 1 [ 1098.299187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_no_route -r 1 [ 1098.333529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_no_route -r 1 [ 1098.362910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_no_route -r 1 [ 1098.387797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_no_route -r 1 [ 1098.415767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_no_route -r 1 [ 1098.443598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_no_route -r 1 [ 1098.467730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1098.641563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.670719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1098.701246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.726490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.752626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1098.782250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.803813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.823437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1098.845945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1098.866662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1098.890376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1098.917704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1098.940420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1098.965764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1098.994573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.021545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1099.445822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_timeout -r 1 [ 1099.478929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_timeout -r 1 [ 1099.518630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_timeout -r 1 [ 1099.555626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_timeout -r 1 [ 1099.614760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_timeout -r 1 [ 1099.645595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_timeout -r 1 [ 1099.673938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_timeout -r 1 [ 1099.710300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_timeout -r 1 [ 1099.742070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1099.761263] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1099.767073] LNet: Skipped 5 previous similar messages [ 1099.933560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.962161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1099.993691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.023418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.043047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1100.071216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.106288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.130570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.182558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.203732] LNet: 61313:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.208412] LNet: 61313:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1100.230551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.264152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.292523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.319435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.352918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.372788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.397385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1100.795858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_error -r 1 [ 1100.820871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_error -r 1 [ 1100.844212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_error -r 1 [ 1100.869948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_error -r 1 [ 1100.921261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_error -r 1 [ 1100.959236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_error -r 1 [ 1101.013840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_error -r 1 [ 1101.096576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_error -r 1 [ 1101.135473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1101.367977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.392553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.417804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.453982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.487674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1101.516567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.553431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.595572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.631326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.667432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.696405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.730578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.758674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.808590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.836370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.872198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1106.855774] LNet: 61514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.860529] LNetError: 61514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.868179] LNet: Removed LNI 192.168.206.54@tcp [ 1106.878142] LNet: Removed LNI 192.168.206.54@tcp1 [ 1107.180047] Key type .llcrypt unregistered [ 1107.190866] Key type ._llcrypt unregistered [ 1120.294981] Key type ._llcrypt registered [ 1120.297267] Key type .llcrypt registered [ 1121.373588] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:09:36 (1789016976) [ 1121.971442] Key type .llcrypt unregistered [ 1121.973168] Key type ._llcrypt unregistered [ 1126.207437] Key type ._llcrypt registered [ 1126.217120] Key type .llcrypt registered [ 1126.403452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1126.418246] alg: No test for adler32 (adler32-zlib) [ 1127.292328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1127.314202] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1132.781723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1133.388588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_dropped -r 1 [ 1133.420946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_dropped -r 1 [ 1133.460564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1133.486327] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1133.699829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.726886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.752567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.779629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.803220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1133.839913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.866234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.913512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.945439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.972617] LNet: 62760:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1134.003698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1134.061621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1134.090152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1134.112509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1134.146345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1134.173555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.200465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1134.780888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_error -r 1 [ 1134.833948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_error -r 1 [ 1134.870573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1134.887601] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1135.122155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1135.153803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1135.181387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1135.202306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1135.230980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1135.263642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1135.314483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.341966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.380218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.398510] LNet: 62871:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1135.419463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.483268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.509934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.540882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.572721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.600368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.639895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1136.176742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_timeout -r 1 [ 1136.211581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_timeout -r 1 [ 1136.243598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1136.262698] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1136.451177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.496314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1136.527286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1136.561322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.593473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1136.614310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.638715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.672656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.708303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.736041] LNet: 62982:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1136.763431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.808242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.838428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.878079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.914483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1136.961355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.010354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1137.267947] LNet: 63024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.276927] LNetError: 63024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.299205] LNet: Removed LNI 192.168.206.54@tcp [ 1137.613599] Key type .llcrypt unregistered [ 1137.615731] Key type ._llcrypt unregistered [ 1150.664640] Key type ._llcrypt registered [ 1150.666467] Key type .llcrypt registered [ 1152.122618] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:10:07 (1789017007) [ 1153.031818] Key type .llcrypt unregistered [ 1153.041106] Key type ._llcrypt unregistered [ 1158.835773] Key type ._llcrypt registered [ 1158.840647] Key type .llcrypt registered [ 1159.099910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1159.123332] alg: No test for adler32 (adler32-zlib) [ 1160.037529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1160.087594] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1168.332987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1177.603864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1177.640800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1177.678239] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1179.026815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_dropped -r 1 [ 1179.072362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e remote_dropped -r 1 [ 1179.101511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_dropped -r 1 [ 1179.149558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e remote_dropped -r 1 [ 1179.203388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e remote_dropped -r 1 [ 1179.297709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e remote_dropped -r 1 [ 1179.366916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e remote_dropped -r 1 [ 1179.451945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e remote_dropped -r 1 [ 1179.491170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1179.529918] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1179.539888] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.154@tcp [ 1179.767245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.794115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.821337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.862880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.900145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1179.924721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.953366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.987117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.011825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.032897] LNet: 64489:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.068297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.103296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.121468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.157069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.180717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.208107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.250835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1180.309588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.182583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_error -r 1 [ 1181.236889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e remote_error -r 1 [ 1181.271683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_error -r 1 [ 1181.304904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e remote_error -r 1 [ 1181.338785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e remote_error -r 1 [ 1181.377216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e remote_error -r 1 [ 1181.412577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e remote_error -r 1 [ 1181.443644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e remote_error -r 1 [ 1181.485650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1181.501972] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1181.504792] LNet: Skipped 2 previous similar messages [ 1181.823039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.870555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.895382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.918977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.942899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1181.984543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.011266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.035598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.057833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.071176] LNet: 64614:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.086770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.109780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.136722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.182878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.206494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.230108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.252927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1182.280374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.750259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_timeout -r 1 [ 1182.799478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e remote_timeout -r 1 [ 1182.853714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_timeout -r 1 [ 1182.906778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e remote_timeout -r 1 [ 1182.959490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e remote_timeout -r 1 [ 1183.001862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e remote_timeout -r 1 [ 1183.050799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e remote_timeout -r 1 [ 1183.080853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e remote_timeout -r 1 [ 1183.111243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1183.122735] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1183.331642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.362506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.407411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.438513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.492516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1183.527739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.558810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.589816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.649321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.670802] LNet: 64740:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.696526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.725955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.748630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.772602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.796669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.845042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.910904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.957628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.154498] LNet: 64818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.181950] LNetError: 64818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.193679] LNet: Removed LNI 192.168.206.54@tcp [ 1189.213034] LNet: Removed LNI 192.168.206.54@tcp1 [ 1189.897665] Key type .llcrypt unregistered [ 1189.900281] Key type ._llcrypt unregistered [ 1206.367828] Key type ._llcrypt registered [ 1206.369596] Key type .llcrypt registered [ 1207.656088] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:11:02 (1789017062) [ 1208.937273] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1210.230916] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:11:05 (1789017065) [ 1210.913709] Key type .llcrypt unregistered [ 1210.916418] Key type ._llcrypt unregistered [ 1215.817457] Key type ._llcrypt registered [ 1215.819801] Key type .llcrypt registered [ 1216.015518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1216.031515] alg: No test for adler32 (adler32-zlib) [ 1216.897787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1216.917364] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1224.103369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1224.911652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e network_timeout -r 1 [ 1224.952808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e network_timeout -r 1 [ 1224.984825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1225.024734] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1225.120303] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1225.370540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.421182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.454488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.492406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.539303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1225.579440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.627753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.675832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.732981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.751522] LNet: 66359:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.767286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1225.800402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1225.838633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.870077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.906403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.942595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.191895] LNet: 66398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.198383] LNetError: 66398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.218671] LNet: Removed LNI 192.168.206.54@tcp [ 1226.694493] Key type .llcrypt unregistered [ 1226.697757] Key type ._llcrypt unregistered [ 1246.074414] Key type ._llcrypt registered [ 1246.076528] Key type .llcrypt registered [ 1246.207258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1246.220077] alg: No test for adler32 (adler32-zlib) [ 1247.059914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1247.088830] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1254.700574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1265.389950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1265.463570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1265.509215] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1267.098631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e network_timeout -r 1 [ 1267.136111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e network_timeout -r 1 [ 1267.171424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e network_timeout -r 1 [ 1267.213826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e network_timeout -r 1 [ 1267.266995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e network_timeout -r 1 [ 1267.340686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e network_timeout -r 1 [ 1267.400672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e network_timeout -r 1 [ 1267.431773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e network_timeout -r 1 [ 1267.463673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1267.485831] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1267.555427] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1267.565167] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.154@tcp [ 1267.740315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1267.773796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1267.807781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1267.837166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1267.864089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1267.897348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1267.940064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1267.973162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1267.994488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1268.008724] LNet: 67276:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1268.020878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1268.045802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1268.066402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1268.087434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1268.109503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.137346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1272.482277] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1272.842624] LNet: 67350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.854073] LNetError: 67350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.864284] LNet: Removed LNI 192.168.206.54@tcp [ 1272.893025] LNet: Removed LNI 192.168.206.54@tcp1 [ 1273.321764] Key type .llcrypt unregistered [ 1273.327415] Key type ._llcrypt unregistered [ 1289.390237] Key type ._llcrypt registered [ 1289.391892] Key type .llcrypt registered [ 1290.398824] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:12:25 (1789017145) [ 1290.822305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1290.832717] alg: No test for adler32 (adler32-zlib) [ 1291.718543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1291.760591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1291.782538] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1292.375771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1292.393120] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1293.229535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1293.320582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1293.358454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1293.394524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1293.429636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1293.469292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1293.502994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.54@tcp [ 1293.534487] LNetError: 68448:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.54@tcp to recovery [ 1293.794403] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1293.797177] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1295.840309] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1296.975416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1299.936469] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1300.644366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1300.683707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1300.722238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1300.738733] LNet: 68555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.745317] LNetError: 68555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.753483] LNet: Removed LNI 192.168.206.54@tcp [ 1300.761752] LNet: Removed LNI 192.168.206.54@tcp1 [ 1300.777491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.830216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1300.856739] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1301.592959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1301.617365] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1302.557485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1302.608964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.653746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1302.690648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1302.726426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1302.777887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1302.821605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1302.854303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.54@tcp [ 1302.873210] LNetError: 68828:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.54@tcp to recovery [ 1302.883114] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1305.243795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1308.000643] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1308.014710] LNet: Skipped 1 previous similar message [ 1313.157205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1313.202768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1313.267279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1319.280502] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:12:54 (1789017174) [ 1319.603872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1319.623696] LNet: 69535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.629745] LNetError: 69535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.636839] LNetError: 69535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1319.641635] LNet: Removed LNI 192.168.206.54@tcp [ 1319.655512] LNet: Removed LNI 192.168.206.54@tcp1 [ 1319.678212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1319.749963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1319.778608] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1320.290743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1320.315731] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1321.027247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1321.085475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1321.112602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1321.164385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1321.215638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1321.277691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1321.331371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.54@tcp --health 0 [ 1321.463337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1321.760153] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.54@tcp [ 1322.784707] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1322.792169] LNet: Skipped 1 previous similar message [ 1336.746991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.947597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1336.996117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1337.043486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.54@tcp --health 500 [ 1337.145218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1343.721158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1343.768717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1343.782730] LNet: 70027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.792520] LNetError: 70027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.801679] LNetError: 70027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1343.813451] LNet: Removed LNI 192.168.206.54@tcp [ 1343.827417] LNet: Removed LNI 192.168.206.54@tcp1 [ 1343.850849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1343.904977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1343.937110] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1344.850724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1344.871269] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1345.771955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1345.816739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1345.890592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1345.919265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1345.963960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1346.020266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1346.058413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1346.093878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.54@tcp --health 0 [ 1348.000725] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1348.010840] LNet: Skipped 3 previous similar messages [ 1348.471500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1358.727350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1358.765997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1358.802655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1366.460927] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:13:41 (1789017221) [ 1366.984262] LNet: 71085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.001367] LNetError: 71085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.019549] LNetError: 71085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1367.034557] LNet: Removed LNI 192.168.206.54@tcp [ 1367.052256] LNet: Removed LNI 192.168.206.54@tcp1 [ 1367.709933] Key type .llcrypt unregistered [ 1367.715183] Key type ._llcrypt unregistered [ 1374.989509] Key type ._llcrypt registered [ 1374.991850] Key type .llcrypt registered [ 1375.404220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1375.442238] alg: No test for adler32 (adler32-zlib) [ 1376.404828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1376.454438] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1385.550727] Lustre: DEBUG MARKER: Initial discovery [ 1385.591538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.154@tcp [ 1387.742764] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1387.963250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1388.013848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.154@tcp [ 1388.037756] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1388.073733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1388.139743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.154@tcp -r 1 -m GET -l 3 [ 1388.187723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.154@tcp -r 1 -m GET -e local_error [ 1388.224918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.154@tcp -r 1 -m PUT -l 6 [ 1388.768165] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1389.426837] Lustre: DEBUG MARKER: Force oleg654-client.virtnet to discover 192.168.206.154@tcp (in background) [ 1389.473325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.154@tcp [ 1390.619271] Lustre: DEBUG MARKER: Force oleg654-server to discover 192.168.206.54@tcp [ 1392.545326] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1392.719994] Lustre: DEBUG MARKER: Wait for 71836 [ 1393.955803] Lustre: DEBUG MARKER: Finished wait on 71836 [ 1394.157861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1394.198624] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1394.207871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1394.465710] LNet: 72295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.474549] LNetError: 72295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.495749] LNet: Removed LNI 192.168.206.54@tcp [ 1394.811607] Key type .llcrypt unregistered [ 1394.817428] Key type ._llcrypt unregistered [ 1408.701081] Key type ._llcrypt registered [ 1408.709569] Key type .llcrypt registered [ 1409.948134] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:14:25 (1789017265) [ 1410.391376] Key type .llcrypt unregistered [ 1410.400037] Key type ._llcrypt unregistered [ 1415.319763] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1415.506054] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1415.513243] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1415.525599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1417.988677] Key type ._llcrypt registered [ 1417.991440] Key type .llcrypt registered [ 1418.170426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1418.182780] alg: No test for adler32 (adler32-zlib) [ 1419.157780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.225580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.271097] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1419.820198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1419.844167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1426.735314] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:14:42 (1789017282) [ 1426.874780] LNet: 74365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.879465] LNetError: 74365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1426.884799] LNet: Removed LNI 10.1.2.3@tcp [ 1426.904860] LNet: Removed LNI 192.168.206.54@tcp [ 1427.440296] Key type .llcrypt unregistered [ 1427.443173] Key type ._llcrypt unregistered [ 1433.560057] Key type ._llcrypt registered [ 1433.562993] Key type .llcrypt registered [ 1433.745734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1433.768992] alg: No test for adler32 (adler32-zlib) [ 1434.752562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.805673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.830341] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1435.373961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1435.396402] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1436.197050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.54@tcp [ 1437.280394] LNetError: Unexpected error -22 connecting to 192.168.206.54@tcp at host 192.168.206.54:988 [ 1441.556221] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:14:57 (1789017297) [ 1441.765205] LNet: 75506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.772566] LNetError: 75506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1441.777852] LNet: Removed LNI 10.1.2.3@tcp [ 1441.793428] LNet: Removed LNI 192.168.206.54@tcp [ 1442.500773] Key type .llcrypt unregistered [ 1442.502473] Key type ._llcrypt unregistered [ 1448.617454] Key type ._llcrypt registered [ 1448.619266] Key type .llcrypt registered [ 1448.832158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1448.839879] alg: No test for adler32 (adler32-zlib) [ 1449.729191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.763980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1449.780542] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1450.420465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1450.447791] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1451.117417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.54@tcp1 --nid 192.168.206.54@tcp2 [ 1455.069055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp1 192.168.206.54@tcp2 [ 1455.095624] LNetError: 76338:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.54@tcp1 to 192.168.206.54@tcp2 - Route table may be misconfigured [ 1455.120488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp2 192.168.206.54@tcp1 [ 1461.147551] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:15:16 (1789017316) [ 1461.406448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.432578] LNet: 76912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.446390] LNetError: 76912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1462.496149] LNet: Removed LNI 192.168.206.54@tcp1 [ 1462.501818] LNetError: 76912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1462.521653] LNet: Removed LNI 192.168.206.54@tcp2 [ 1462.609855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.735157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.769863] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1463.590993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1463.616149] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1464.509315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1464.538731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.54@tcp [ 1464.574481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp1 192.168.206.54@tcp1 [ 1464.612724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.54@tcp -d 192.168.206.54@tcp -e network_timeout [ 1464.658742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -e network_timeout [ 1464.701861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -e network_timeout [ 1464.746992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -e network_timeout [ 1464.794461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1464.821144] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1464.852737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1464.939359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1465.018211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1465.761741] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1471.029342] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:15:26 (1789017326) [ 1471.212984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.236599] LNet: 77774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.245326] LNetError: 77774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.288962] LNet: Removed LNI 192.168.206.54@tcp [ 1473.320779] LNet: Removed LNI 192.168.206.54@tcp1 [ 1473.387107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.509762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1473.787807] LNet: 77811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.146262] Key type .llcrypt unregistered [ 1474.154918] Key type ._llcrypt unregistered [ 1485.711763] Key type ._llcrypt registered [ 1485.715255] Key type .llcrypt registered [ 1486.914654] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:15:42 (1789017342) [ 1487.360600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1487.371474] alg: No test for adler32 (adler32-zlib) [ 1488.274905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1489.550981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1491.765801] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:15:46 (1789017346) [ 1492.000804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.029427] LNet: 78906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.048954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.112760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1492.148786] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1492.889232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1492.907686] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1493.727854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1493.759572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp1 [ 1493.791224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp1 [ 1499.934755] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:15:55 (1789017355) [ 1501.734851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg654-server [ 1503.693492] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:15:58 (1789017358) [ 1505.580369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg654-server [ 1507.362871] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:16:02 (1789017362) [ 1509.692238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg654-server [ 1511.481744] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:16:06 (1789017366) [ 1513.387929] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg654-server [ 1514.898332] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:16:10 (1789017370) [ 1516.762660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg654-server [ 1518.855868] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:16:13 (1789017373) [ 1520.628761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg654-server [ 1522.388893] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:16:17 (1789017377) [ 1524.588982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg654-server [ 1526.648798] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:16:21 (1789017381) [ 1528.657578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg654-server [ 1530.987749] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:16:25 (1789017385) [ 1533.313441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg654-server [ 1535.373604] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:16:30 (1789017390) [ 1535.479840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.507703] LNet: 82775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.521811] LNetError: 82775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.531382] LNet: Removed LNI 192.168.206.54@tcp [ 1536.561626] LNet: Removed LNI 192.168.206.54@tcp1 [ 1536.590411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.666268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1536.685275] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1537.522460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1537.599243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1537.803220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.834925] LNet: 82930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1537.846384] LNetError: 82930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.857630] LNetError: 82930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1537.880886] LNet: Removed LNI 192.168.206.54@tcp [ 1537.901786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.013097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1538.047233] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1539.019168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1539.312513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.358390] LNetError: 83075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.389566] LNet: Removed LNI 192.168.206.54@tcp [ 1539.412823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.468869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.492169] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1540.419528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1540.518566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1540.812456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.838264] LNet: 83222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.845269] LNet: 83222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1541.866512] LNet: Removed LNI 192.168.206.54@tcp [ 1541.918698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.970983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1542.000384] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1542.844423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1543.014853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.033919] LNetError: 83368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.040062] LNetError: 83368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1543.058868] LNet: Removed LNI 192.168.206.54@tcp [ 1543.099479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.204982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.262963] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1544.059332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1544.111156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1544.356958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.449172] LNet: Removed LNI 192.168.206.54@tcp [ 1545.484399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.564535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1545.605661] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1546.349300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1546.543349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.556445] LNet: 83660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.562430] LNet: 83660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1547.625640] LNet: Removed LNI 192.168.206.54@tcp [ 1547.693922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.772684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1547.793906] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1548.576250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1548.664418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1548.891922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.916336] LNetError: 83808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.922373] LNetError: 83808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1549.998248] LNet: Removed LNI 192.168.206.54@tcp [ 1550.027302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.086089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1550.105565] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1550.680472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1550.842641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.912604] LNet: Removed LNI 192.168.206.54@tcp [ 1551.948080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.010604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1552.044484] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1552.645509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1552.733714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1552.895094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.959756] LNet: Removed LNI 192.168.206.54@tcp [ 1554.003615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.071197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1554.097721] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1554.789521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1555.064226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.093622] LNet: 84245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.103124] LNet: 84245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1556.143329] LNet: Removed LNI 192.168.206.54@tcp [ 1556.173189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.216517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1556.234547] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1556.841672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1556.899104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1557.127341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.153592] LNetError: 84393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.162352] LNetError: 84393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1557.180351] LNet: Removed LNI 192.168.206.54@tcp [ 1557.212252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.247861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1557.271647] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1557.755922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1558.013504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.083788] LNet: Removed LNI 192.168.206.54@tcp [ 1559.130384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.201370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1559.231769] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1559.774904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1559.831419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.864793] LNet: Removed LNI 192.168.206.54@tcp [ 1559.909797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.955693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1559.977869] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1560.867632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1560.933415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.954931] LNet: Removed LNI 192.168.206.54@tcp [ 1560.982935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.057766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1561.159178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.220320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.288721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1561.381864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.418223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.482891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1561.587731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.631928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.708303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1561.731319] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1562.429772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.451901] LNet: Removed LNI 192.168.206.54@tcp [ 1562.468740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.534545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1567.813630] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:17:03 (1789017423) [ 1567.900396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.939834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.008494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1568.032249] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/180] [ 1569.251484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.296633] LNet: Removed LNI 192.168.206.54@tcp231 [ 1569.325765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.399960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1569.417501] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/180] [ 1569.422760] LNet: 85704:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1569.830469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1569.863185] LNet: Removed LNI 192.168.206.54@tcp231 [ 1569.876872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1569.901876] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/180] [ 1570.786608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.813161] LNet: Removed LNI 192.168.206.54@tcp231 [ 1570.839392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.884316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1570.904053] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/0] [ 1570.910457] LNet: 85857:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1571.160712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1571.198408] LNet: Removed LNI 192.168.206.54@tcp231 [ 1571.208983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1571.236198] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/0] [ 1572.255239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.283409] LNet: 86000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.290139] LNet: 86000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1572.319741] LNet: Removed LNI 192.168.206.54@tcp231 [ 1572.355546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.428725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1572.447801] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/60] [ 1572.451971] LNet: 86008:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1572.877447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1572.910523] LNet: Removed LNI 192.168.206.54@tcp231 [ 1572.931125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1572.991622] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/60] [ 1573.819022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.833753] LNetError: 86151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.838889] LNetError: 86151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1573.854770] LNet: Removed LNI 192.168.206.54@tcp231 [ 1573.885711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.958633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1573.978884] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/180] [ 1574.379173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1574.430654] LNet: Removed LNI 192.168.206.54@tcp231 [ 1574.462286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1574.499858] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/180] [ 1581.334217] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:17:16 (1789017436) [ 1581.473047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.533199] LNet: Removed LNI 192.168.206.54@tcp231 [ 1581.560586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.616468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1581.642513] LNet: Added LNI 192.168.206.54@tcp232 [8/256/0/180] [ 1582.429454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1582.461865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1582.488779] LNet: Removed LNI 192.168.206.54@tcp232 [ 1582.510473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1582.547406] LNet: Added LNI 192.168.206.54@tcp232 [8/256/0/180] [ 1583.385988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1583.428909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1583.460438] LNet: Removed LNI 192.168.206.54@tcp232 [ 1583.492126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1583.514475] LNet: Added LNI 192.168.206.54@tcp232 [8/256/0/180] [ 1584.357576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1584.395850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1584.420554] LNet: Removed LNI 192.168.206.54@tcp232 [ 1584.433113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1584.456945] LNet: Added LNI 192.168.206.54@tcp232 [8/256/0/180] [ 1585.095552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1585.140197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1585.172965] LNet: Removed LNI 192.168.206.54@tcp232 [ 1585.200240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1590.819882] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:17:26 (1789017446) [ 1591.784836] Key type .llcrypt unregistered [ 1591.790606] Key type ._llcrypt unregistered [ 1597.514191] Key type ._llcrypt registered [ 1597.520546] Key type .llcrypt registered [ 1597.789546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1597.798526] alg: No test for adler32 (adler32-zlib) [ 1598.681582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1598.718067] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1605.768671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1615.037486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.102574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1615.150671] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1616.467814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_interrupt -r 2 [ 1616.498573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_interrupt -r 2 [ 1616.526530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_interrupt -r 2 [ 1616.562306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_interrupt -r 2 [ 1616.595176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_interrupt -r 2 [ 1616.632467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_interrupt -r 2 [ 1616.667336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_interrupt -r 2 [ 1616.705750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_interrupt -r 2 [ 1616.741330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1616.769810] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1616.776525] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1616.999467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1617.021884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1617.046739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1617.072661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1617.096475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1617.116428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1617.138751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.161925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.186257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1617.201601] LNet: 88657:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1617.214781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1617.236823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1617.265557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1617.285664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1617.309480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1617.338462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1617.745746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_dropped -r 2 [ 1617.772938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_dropped -r 2 [ 1617.796907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_dropped -r 2 [ 1617.826406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_dropped -r 2 [ 1617.856741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_dropped -r 2 [ 1617.887241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_dropped -r 2 [ 1617.915893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_dropped -r 2 [ 1617.939437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_dropped -r 2 [ 1617.965642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1617.981628] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1618.189765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1618.217441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1618.255525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1618.303223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1618.341111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1618.382757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1618.408244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1618.444381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1618.489572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1618.523430] LNet: 88780:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1618.542360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1618.568849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1618.595259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1618.627660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1618.680082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.715001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1619.132899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_aborted -r 2 [ 1619.183653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_aborted -r 2 [ 1619.234096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_aborted -r 2 [ 1619.273839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_aborted -r 2 [ 1619.309938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_aborted -r 2 [ 1619.345995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_aborted -r 2 [ 1619.371991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_aborted -r 2 [ 1619.399528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_aborted -r 2 [ 1619.428336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1619.449435] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1619.737318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1619.780336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1619.821269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1619.848669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1619.875071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1619.901689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1619.939720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1619.978277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1620.005956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1620.022044] LNet: 88902:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1620.036448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1620.061690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1620.095928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1620.122829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1620.157356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1620.205824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1620.701507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_no_route -r 2 [ 1620.735750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_no_route -r 2 [ 1620.772791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_no_route -r 2 [ 1620.811631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_no_route -r 2 [ 1620.854505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_no_route -r 2 [ 1620.894348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_no_route -r 2 [ 1620.931877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_no_route -r 2 [ 1620.977472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_no_route -r 2 [ 1621.050879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1621.264924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1621.290441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1621.318465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1621.341506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1621.364953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1621.408473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1621.436347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1621.477099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1621.512852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1621.553774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1621.618682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1621.664993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1621.713993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1621.751927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1621.808814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1622.450721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_timeout -r 2 [ 1622.486905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_timeout -r 2 [ 1622.521241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_timeout -r 2 [ 1622.563597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_timeout -r 2 [ 1622.609653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_timeout -r 2 [ 1622.650793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_timeout -r 2 [ 1622.684868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_timeout -r 2 [ 1622.739493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_timeout -r 2 [ 1622.773742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1622.792762] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1622.797567] LNet: Skipped 1 previous similar message [ 1623.069499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1623.097731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1623.123329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1623.153767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1623.196083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1623.232403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1623.276013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1623.303090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1623.326399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1623.345720] LNet: 89146:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1623.357604] LNet: 89146:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1623.373934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1623.403123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1623.436466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1623.467535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1623.511356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1623.550857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1624.147463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_dropped -r 2 [ 1624.171906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e remote_dropped -r 2 [ 1624.202918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_dropped -r 2 [ 1624.238474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e remote_dropped -r 2 [ 1624.268472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e remote_dropped -r 2 [ 1624.296788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e remote_dropped -r 2 [ 1624.331469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e remote_dropped -r 2 [ 1624.375688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e remote_dropped -r 2 [ 1624.449569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1624.492226] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.154@tcp [ 1624.748810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1624.795674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1624.841669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1624.872505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1624.933507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1624.963753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1625.001863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1625.043632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1625.085357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1625.127629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1625.172599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1625.210454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1625.250728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1625.298625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1625.326881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1629.944431] LNet: 89343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.962760] LNetError: 89343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.974576] LNet: Removed LNI 192.168.206.54@tcp [ 1629.990101] LNet: Removed LNI 192.168.206.54@tcp1 [ 1630.364752] Key type .llcrypt unregistered [ 1630.368552] Key type ._llcrypt unregistered [ 1646.009941] Key type ._llcrypt registered [ 1646.013333] Key type .llcrypt registered [ 1647.163804] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:18:22 (1789017502) [ 1647.645421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1647.659386] alg: No test for adler32 (adler32-zlib) [ 1648.560293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.626050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1648.685962] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1649.946966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1649.991345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.54@tcp [ 1650.029372] LNetError: 90320:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.54@tcp to recovery [ 1650.151307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.54@tcp [ 1650.331017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1658.879398] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:18:32 (1789017512) [ 1659.092715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.183127] LNet: 90904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.201702] LNetError: 90904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.248443] LNet: Removed LNI 192.168.206.54@tcp [ 1659.284774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.414192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1659.470234] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1660.494544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1660.525469] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1661.805805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.54@tcp [ 1661.826916] LNetError: 91177:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.54@tcp to recovery [ 1662.432752] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1662.433980] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1663.135868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.54@tcp [ 1672.970965] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:18:47 (1789017527) [ 1675.075051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg654-server [ 1677.517690] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:18:52 (1789017532) [ 1680.133880] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg654-server [ 1682.545860] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:18:57 (1789017537) [ 1685.208849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg654-server [ 1687.408402] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:19:02 (1789017542) [ 1687.587886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.641578] LNet: 92801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.654879] LNetError: 92801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.672135] LNet: Removed LNI 192.168.206.54@tcp [ 1687.709341] LNet: Removed LNI 192.168.206.54@tcp1 [ 1687.793762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.023337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.073381] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1689.326271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1689.423199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1689.503603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1689.620553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1689.695341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1689.756315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1697.365688] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:19:12 (1789017552) [ 1697.472258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.508841] LNet: 93530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.516313] LNetError: 93530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.522264] LNetError: 93530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1697.544863] LNet: Removed LNI 192.168.206.54@tcp [ 1697.566389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.631506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.679798] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1706.123255] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:19:21 (1789017561) [ 1706.301621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.323745] LNet: 94239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.331681] LNetError: 94239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.348095] LNet: Removed LNI 192.168.206.54@tcp [ 1706.378326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.509058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1706.559788] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1714.321680] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:19:29 (1789017569) [ 1715.838938] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1718.135495] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:19:32 (1789017572) [ 1718.706392] LNet: 95278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.712944] LNetError: 95278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.735496] LNet: Removed LNI 192.168.206.54@tcp [ 1719.567671] Key type .llcrypt unregistered [ 1719.574033] Key type ._llcrypt unregistered [ 1726.497529] Key type ._llcrypt registered [ 1726.500594] Key type .llcrypt registered [ 1726.823842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1726.846862] alg: No test for adler32 (adler32-zlib) [ 1727.843367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1727.884801] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1735.607313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1741.568586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.154@tcp [ 1742.024702] LNet: 95693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.032373] LNetError: 95693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.058358] LNet: Removed LNI 192.168.206.54@tcp [ 1742.567843] Key type .llcrypt unregistered [ 1742.570707] Key type ._llcrypt unregistered [ 1754.942429] Key type ._llcrypt registered [ 1754.944575] Key type .llcrypt registered [ 1756.479373] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1758.305479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1760.028243] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:20:15 (1789017615) [ 1761.603842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1763.703400] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:20:18 (1789017618) [ 1766.369259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg654-server [ 1767.983331] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:20:23 (1789017623) [ 1769.815286] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg654-server [ 1772.000866] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:20:27 (1789017627) [ 1773.435656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.607963] alg: No test for adler32 (adler32-zlib) [ 1774.696238] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1774.928187] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1776.640860] Key type lgssc registered [ 1777.829606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1800.201958] Key type lgssc unregistered [ 1800.480114] LNet: 98866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.484702] LNetError: 98866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.504910] LNet: Removed LNI 192.168.206.54@tcp [ 1801.351986] Key type .llcrypt unregistered [ 1801.359453] Key type ._llcrypt unregistered [ 1817.051361] Key type ._llcrypt registered [ 1817.053150] Key type .llcrypt registered [ 1818.456510] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:21:13 (1789017673) [ 1819.019855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.043251] alg: No test for adler32 (adler32-zlib) [ 1820.121417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.221567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1820.273889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1820.318856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1820.405814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1827.218670] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:21:22 (1789017682) [ 1827.450680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.487438] LNet: 100279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.508939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.582731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.622819] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1829.260987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.325219] LNet: 100424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.344315] LNetError: 100424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.367827] LNet: Removed LNI 192.168.206.54@tcp [ 1829.411828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.545832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1829.601168] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1830.250747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.265738] LNetError: 100452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.290087] LNet: Removed LNI 192.168.206.54@tcp [ 1830.322202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.375917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1830.400989] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1839.394887] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:21:34 (1789017694) [ 1839.575179] LNet: 101043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.579381] LNet: 101043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1839.584542] LNetError: 101043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.596308] LNet: Removed LNI 192.168.206.54@tcp [ 1840.615483] Key type .llcrypt unregistered [ 1840.617224] Key type ._llcrypt unregistered [ 1847.001446] Key type ._llcrypt registered [ 1847.005235] Key type .llcrypt registered [ 1847.329748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1847.352695] alg: No test for adler32 (adler32-zlib) [ 1848.321447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.395764] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1848.399531] LNet: *** cfs_fail_loc=e003, val=100*** [ 1849.276427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1849.304742] LNet: *** cfs_fail_loc=e003, val=100*** [ 1849.307093] LNet: Skipped 2 previous similar messages [ 1849.312687] Ping reply received with mapping: [ 1849.315360] found metadata reply NI[192.168.206.54@tcp] [ 1849.318123] 36 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1849.321358] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1849.324034] 5a 5a 5a 5a 5a 5a 5a 5a [ 1849.469855] LNet: 101438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.481384] LNetError: 101438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.537606] LNet: Removed LNI 192.168.206.54@tcp [ 1851.052403] Key type .llcrypt unregistered [ 1851.054875] Key type ._llcrypt unregistered [ 1861.560766] Key type ._llcrypt registered [ 1861.562544] Key type .llcrypt registered [ 1862.966177] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:21:57 (1789017717) [ 1863.749841] Key type .llcrypt unregistered [ 1863.755664] Key type ._llcrypt unregistered [ 1871.371167] Key type ._llcrypt registered [ 1871.374841] Key type .llcrypt registered [ 1871.561658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1871.573726] alg: No test for adler32 (adler32-zlib) [ 1872.819246] LNetError: 102475:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1873.250814] Key type .llcrypt unregistered [ 1873.257677] Key type ._llcrypt unregistered [ 1878.575726] Key type ._llcrypt registered [ 1878.579128] Key type .llcrypt registered [ 1880.119416] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:22:15 (1789017735) [ 1880.682737] Key type .llcrypt unregistered [ 1880.689787] Key type ._llcrypt unregistered [ 1887.757892] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1887.765862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1890.450616] Key type ._llcrypt registered [ 1890.453499] Key type .llcrypt registered [ 1890.692286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1890.718656] alg: No test for adler32 (adler32-zlib) [ 1891.657241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.725827] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1901.861838] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:22:36 (1789017756) [ 1902.086992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1902.109679] LNet: 104249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.115818] LNetError: 104249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1902.133635] LNet: Removed LNI 10.1.2.3@tcp [ 1902.152840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.874394] Key type .llcrypt unregistered [ 1902.878644] Key type ._llcrypt unregistered [ 1909.637315] Key type ._llcrypt registered [ 1909.640292] Key type .llcrypt registered [ 1909.913121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1909.951913] alg: No test for adler32 (adler32-zlib) [ 1910.956077] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1911.084469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1911.337723] LNet: 104535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.347828] LNetError: 104535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.377399] LNet: Removed LNI 192.168.206.54@tcp [ 1912.391388] Key type .llcrypt unregistered [ 1912.393254] Key type ._llcrypt unregistered [ 1925.483957] Key type ._llcrypt registered [ 1925.487324] Key type .llcrypt registered [ 1927.033182] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:23:02 (1789017782) [ 1927.889559] Key type .llcrypt unregistered [ 1927.893077] Key type ._llcrypt unregistered [ 1934.568350] Key type ._llcrypt registered [ 1934.571492] Key type .llcrypt registered [ 1934.804874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1934.845330] alg: No test for adler32 (adler32-zlib) [ 1942.171465] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:23:17 (1789017797) [ 1942.287761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1942.328731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.413374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1942.461037] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1943.112788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.145300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1943.195818] LNet: Removed LNI 192.168.206.54@tcp [ 1943.515196] LNet: 106275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.043659] Key type .llcrypt unregistered [ 1944.047638] Key type ._llcrypt unregistered [ 1955.976196] Key type ._llcrypt registered [ 1955.977817] Key type .llcrypt registered [ 1957.635487] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:23:32 (1789017812) [ 1958.035634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.048760] alg: No test for adler32 (adler32-zlib) [ 1959.004353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.078376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1959.103299] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1959.880432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1959.945320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.54@tcp [ 1959.982812] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1959.993251] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1959.997128] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1959, 0, 0 [ 1960.004946] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1960.007027] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1960.012432] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.023625] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1959, 0, 0 [ 1960.029040] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1960.036480] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1960.040364] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.044932] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1959, 0, 2009 [ 1960.049832] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1960.051806] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1960.055378] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.058275] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1959, 0, 2009 [ 1960.061729] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1966.128470] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:23:41 (1789017821) [ 1966.637549] LNet: 107812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.650256] LNetError: 107812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.664854] LNet: Removed LNI 192.168.206.54@tcp [ 1967.440436] Key type .llcrypt unregistered [ 1967.442179] Key type ._llcrypt unregistered [ 1973.777833] Key type ._llcrypt registered [ 1973.780148] Key type .llcrypt registered [ 1973.949650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1973.982919] alg: No test for adler32 (adler32-zlib) [ 1974.962327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1975.021580] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1982.092832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1990.521544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.563633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1990.601803] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1991.458322] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1991.690241] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1991.698589] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1991.701668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1994.023826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1994.043318] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2004.727924] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:24:19 (1789017859) [ 2004.944939] LNet: 109312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.964187] LNetError: 109312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.975577] LNet: Removed LNI 192.168.206.54@tcp [ 2006.048589] LNet: Removed LNI 192.168.206.54@tcp1 [ 2006.052743] LNetError: 109312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2006.059468] LNetError: 109312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2006.078324] LNet: Removed LNI 10.1.2.3@tcp99 [ 2006.698760] Key type .llcrypt unregistered [ 2006.705687] Key type ._llcrypt unregistered [ 2012.970084] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2012.980884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2015.707790] Key type ._llcrypt registered [ 2015.709479] Key type .llcrypt registered [ 2015.831828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2015.845521] alg: No test for adler32 (adler32-zlib) [ 2016.802690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.851596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2016.868822] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2017.594353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2017.612033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2019.611265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2019.661377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2019.701368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2019.734596] LNetError: 110059:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2019.780637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2019.836319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2020.154273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2020.232811] LNet: 110079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.245065] LNetError: 110079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2020.253685] LNet: Removed LNI 10.1.2.3@tcp [ 2021.287258] LNet: Removed LNI 192.168.206.54@tcp [ 2021.936476] Key type .llcrypt unregistered [ 2021.938321] Key type ._llcrypt unregistered [ 2032.585669] Key type ._llcrypt registered [ 2032.589745] Key type .llcrypt registered [ 2033.830710] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:24:49 (1789017889) [ 2034.307207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2034.324356] alg: No test for adler32 (adler32-zlib) [ 2035.260702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.319732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2035.340707] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2036.238153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2036.329846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg654-client@tcp [ 2043.715754] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:24:58 (1789017898) [ 2044.006275] LNet: 111608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.034778] LNetError: 111608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2044.108792] LNet: Removed LNI 192.168.206.54@tcp [ 2045.032353] Key type .llcrypt unregistered [ 2045.035406] Key type ._llcrypt unregistered [ 2053.042758] Key type ._llcrypt registered [ 2053.045634] Key type .llcrypt registered [ 2053.238551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.259552] alg: No test for adler32 (adler32-zlib) [ 2054.140343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2054.160586] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2054.192350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2059.890760] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:25:15 (1789017915) [ 2061.106221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2062.658428] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:25:18 (1789017918) [ 2064.252363] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2066.146632] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:25:21 (1789017921) [ 2066.237769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2066.256492] LNet: 113025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.267077] LNetError: 113025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.285901] LNet: Removed LNI 192.168.206.54@tcp [ 2066.330231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2066.452604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2066.488255] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2067.403319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2067.452272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2073.860254] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:25:29 (1789017929) [ 2073.934993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2073.950216] LNet: 113735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.965708] LNetError: 113735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.987798] LNet: Removed LNI 192.168.206.54@tcp [ 2074.016885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.077632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2074.113759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2074.729737] Key type .llcrypt unregistered [ 2074.735141] Key type ._llcrypt unregistered [ 2085.476279] Key type ._llcrypt registered [ 2085.479281] Key type .llcrypt registered [ 2086.599502] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:25:42 (1789017942) [ 2087.218235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2087.231149] alg: No test for adler32 (adler32-zlib) [ 2088.122327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.171436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2088.190151] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2088.771818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2088.820535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2095.515601] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:25:50 (1789017950) [ 2095.609695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2095.650057] LNet: 115285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2095.658134] LNetError: 115285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2095.673063] LNet: Removed LNI 192.168.206.54@tcp [ 2095.721761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.786699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2095.851782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2103.147226] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:25:58 (1789017958) [ 2103.318170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2103.345540] LNet: 115870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.376321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.448781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2103.735737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2103.756953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2103.793072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2104.016546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2104.036451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2104.082096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.101980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.139629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2104.364449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2104.410362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2104.497299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2104.750983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2104.773348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2110.949868] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:26:06 (1789017966) [ 2111.419778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.477849] LNet: 116508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.488695] LNet: 116508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2111.536994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.632975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2111.669798] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2112.566761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2112.600971] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2113.528176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2113.574504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2113.729625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.752353] LNet: 116787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.758655] LNetError: 116787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.765680] LNet: Removed LNI 192.168.206.54@tcp [ 2113.775642] LNet: Removed LNI 192.168.206.54@tcp2 [ 2113.817343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.870996] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2113.885198] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2113.889759] LNet: 116793:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2114.388783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2114.596690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.680218] LNet: Removed LNI 192.168.206.54@tcp [ 2115.682282] LNetError: 116830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.691461] LNetError: 116830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2115.720734] LNet: Removed LNI 192.168.206.54@tcp2 [ 2115.742309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.789381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2115.808040] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2116.482669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2116.506626] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2117.246129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2117.283782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2117.333881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2117.546666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.624236] LNet: Removed LNI 192.168.206.54@tcp [ 2118.625725] LNetError: 117115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.639850] LNetError: 117115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2118.662541] LNet: Removed LNI 192.168.206.54@tcp2 [ 2118.722298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.828965] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2118.835826] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2118.841284] LNet: 117121:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2119.496792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2119.719305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2119.746384] LNet: 117158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.751866] LNet: 117158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2119.756715] LNet: Removed LNI 192.168.206.54@tcp [ 2120.818797] LNet: Removed LNI 192.168.206.54@tcp2 [ 2120.865308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.931727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2120.963249] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2121.908719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2121.942673] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2123.066589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2123.115877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2123.301582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.327874] LNetError: 117437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.332177] LNetError: 117437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2123.339358] LNet: Removed LNI 192.168.206.54@tcp [ 2123.352872] LNet: Removed LNI 192.168.206.54@tcp2 [ 2123.371614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.415621] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2123.419808] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2123.426573] LNet: 117443:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2123.944478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2124.145663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.216249] LNet: Removed LNI 192.168.206.54@tcp [ 2125.239652] LNet: Removed LNI 192.168.206.54@tcp2 [ 2125.285703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.367205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.388328] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2126.039961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2126.076508] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2126.947790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2126.995207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2127.035444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2127.186666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.224233] LNet: Removed LNI 192.168.206.54@tcp [ 2129.257879] LNet: Removed LNI 192.168.206.54@tcp2 [ 2129.285283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.343624] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2129.350119] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2129.354731] LNet: 117772:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2129.686862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2129.926869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.945713] LNet: 117809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.952717] LNet: 117809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2129.958292] LNet: Removed LNI 192.168.206.54@tcp [ 2131.000048] LNet: Removed LNI 192.168.206.54@tcp2 [ 2131.095028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.172179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2131.213846] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2132.162695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2132.183954] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2133.153161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2133.205583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2133.264633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2133.717200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.745311] LNetError: 118106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.752584] LNetError: 118106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2133.766350] LNet: Removed LNI 192.168.206.54@tcp [ 2133.784263] LNet: Removed LNI 192.168.206.54@tcp2 [ 2133.820590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.937220] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2133.959230] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2133.978996] LNet: 118112:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2134.527394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2134.940331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.032144] LNet: Removed LNI 192.168.206.54@tcp [ 2136.045132] LNet: Removed LNI 192.168.206.54@tcp2 [ 2136.075887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.141557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2136.186689] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2136.929195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2136.944606] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2137.806277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2137.850329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2137.877905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2137.953629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2138.341907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.424226] LNet: Removed LNI 192.168.206.54@tcp [ 2140.469053] LNet: Removed LNI 192.168.206.54@tcp2 [ 2140.531265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.600101] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2140.629354] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2141.237706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2148.050871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2149.679275] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:26:44 (1789018004) [ 2149.787976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.819621] LNet: 119225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.824540] LNet: 119225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2149.830730] LNetError: 119225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2149.833908] LNetError: 119225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2149.837865] LNet: Removed LNI 192.168.206.54@tcp [ 2150.893162] LNet: Removed LNI 192.168.206.54@tcp2 [ 2150.963255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.373957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2151.434271] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2152.195266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2152.218557] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2153.188901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.220495] LNet: Removed LNI 10.1.2.3@tcp [ 2153.237121] LNet: Removed LNI 192.168.206.54@tcp [ 2153.260758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.342739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2153.394097] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2153.401753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2154.096977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2154.132945] LNet: Removed LNI 192.168.206.54@tcp [ 2154.143751] LNet: Removed LNI 10.1.2.3@tcp [ 2161.501377] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:26:56 (1789018016) [ 2161.638568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.660377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.707526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.739150] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2162.444427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2162.475882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2162.555692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2162.616391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2162.664942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2162.752903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2162.855762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2162.939395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2163.013508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2163.116719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2163.213070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2163.323664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2163.629579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2170.638712] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 01:27:05 (1789018025) [ 2170.769623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2171.882546] LNet: Removed LNI 192.168.206.54@tcp [ 2171.919727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.068817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2172.143296] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2173.699038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2173.770333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2173.838248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2173.883514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2173.941275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2173.989478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.050558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.102801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.151299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.214490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.271186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.334819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.532814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2174.603760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.665825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.722854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.779549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.826865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.871557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.926474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2174.975582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2175.013445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2175.060909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2180.330596] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:27:15 (1789018035) [ 2180.500294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.544534] LNet: Removed LNI 192.168.206.54@tcp [ 2181.588694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.767524] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2181.900515] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2181.917211] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2181.921943] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2184.257612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2184.283916] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2184.860464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2184.884325] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2186.474933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2186.503995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2186.551894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.54@tcp -m PUT -e local_timeout -r 1 [ 2186.626872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2186.672633] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2186.683509] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2186.939796] LNetError: 121671:0:(lib-move.c:4794:lnet_parse()) 192.168.206.54@tcp, src 192.168.206.54@tcp: Dropping PUT (error -108 looking up sender) [ 2187.937073] LNet: 122129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.947083] LNet: 122129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2187.957229] LNetError: 122129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2187.966955] LNetError: 122129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2187.971572] LNet: Removed LNI 10.1.2.3@tcp [ 2189.042685] LNet: Removed LNI 192.168.206.54@tcp [ 2189.410678] Key type .llcrypt unregistered [ 2189.413206] Key type ._llcrypt unregistered [ 2200.031216] Key type ._llcrypt registered [ 2200.034163] Key type .llcrypt registered [ 2201.152545] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:27:36 (1789018056) [ 2201.629533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.639336] alg: No test for adler32 (adler32-zlib) [ 2202.550851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.953957] LNet: 122980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.457582] Key type .llcrypt unregistered [ 2203.459363] Key type ._llcrypt unregistered [ 2210.340944] Key type ._llcrypt registered [ 2210.343289] Key type .llcrypt registered [ 2210.711643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2210.730363] alg: No test for adler32 (adler32-zlib) [ 2211.765763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2211.810169] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2218.634529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 2218.854145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2219.538798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.154@tcp [ 2219.879421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2219.932995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2219.968192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2220.000970] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.154@tcp [ 2220.005812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2220.038403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2220.085921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2220.121783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2220.163194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2220.204753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2220.235174] LNet: 123429:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2220.255719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2220.288139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2220.329588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2220.370908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2220.406545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2220.441591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2221.116987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2221.302824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2221.327269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2221.359478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2221.384732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2221.426370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2221.451726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2221.493580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2221.519435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2221.546449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2221.559942] LNet: 123548:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2221.573222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2221.596973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2221.621597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2221.642527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2221.666550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2221.694987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2222.230757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.154@tcp [ 2222.460969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2222.481752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2222.503235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2222.523458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2222.547971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2222.576597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2222.602383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2222.621400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2222.640283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2222.659711] LNet: 123667:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2222.689141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2222.740529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2222.779809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2222.810536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2222.837554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2222.887229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2223.519693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2223.779772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2223.820266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2223.845361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2223.882600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2223.922858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2223.958262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2223.993922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2224.041690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2224.092501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2224.151815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2224.221688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2224.276684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2224.322751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2224.383304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2224.418985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2224.672725] LNet: 123833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.677639] LNetError: 123833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.697478] LNet: Removed LNI 192.168.206.54@tcp [ 2225.063863] Key type .llcrypt unregistered [ 2225.065506] Key type ._llcrypt unregistered [ 2241.112992] Key type ._llcrypt registered [ 2241.115371] Key type .llcrypt registered [ 2242.380795] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:28:17 (1789018097) [ 2242.825821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.842399] alg: No test for adler32 (adler32-zlib) [ 2243.803684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.133646] LNet: 124718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.746482] Key type .llcrypt unregistered [ 2244.749427] Key type ._llcrypt unregistered [ 2251.098614] Key type ._llcrypt registered [ 2251.103983] Key type .llcrypt registered [ 2251.340606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2251.364832] alg: No test for adler32 (adler32-zlib) [ 2252.282434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2252.315529] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2259.652248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 2268.339660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.397683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2268.441573] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 2269.584543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2270.292331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.154@tcp [ 2270.523572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2270.564799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2270.592423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2270.619573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2270.639437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2270.661347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2270.691558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2270.720453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2270.746102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2270.752111] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.154@tcp [ 2270.768050] LNet: 125370:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2270.796426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2270.822273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2270.856566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2270.909204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2270.973716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2271.015670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2271.790334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2272.049955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2272.088327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2272.120370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2272.155442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2272.196980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2272.235759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2272.267947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2272.305681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2272.337893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2272.362051] LNet: 125489:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2272.396402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2272.424816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2272.471672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2272.500378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2272.524648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2272.549784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2273.078113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.154@tcp [ 2273.264569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2273.289519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2273.313978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2273.339248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2273.366871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2273.387129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2273.407644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2273.428897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2273.451490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2273.464142] LNet: 125608:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2273.482538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2273.524064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2273.582776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2273.624436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2273.663342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2273.700383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2274.323931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2274.554084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2274.579625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2274.609763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2274.640181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2274.658968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2274.692722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2274.722933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2274.748673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2274.772442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2274.806285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2274.845923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2274.889330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2274.931157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2274.968671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2274.992147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2279.090504] LNet: 125808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.103774] LNetError: 125808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.116666] LNet: Removed LNI 192.168.206.54@tcp [ 2279.127941] LNet: Removed LNI 192.168.206.54@tcp1 [ 2279.518641] Key type .llcrypt unregistered [ 2279.524856] Key type ._llcrypt unregistered [ 2297.385996] Key type ._llcrypt registered [ 2297.388508] Key type .llcrypt registered [ 2298.644728] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 01:29:13 (1789018153) [ 2299.018400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.034991] alg: No test for adler32 (adler32-zlib) [ 2299.973816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.039144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.070696] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2300.843134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2300.859847] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2301.709590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2301.836679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2302.880221] LNet: Removed LNI 192.168.206.54@tcp [ 2308.647468] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:29:23 (1789018163) [ 2310.160066] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2312.575623] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:29:27 (1789018167) [ 2321.249743] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:29:36 (1789018176) [ 2329.803569] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:29:44 (1789018184) [ 2337.407681] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:29:52 (1789018192) [ 2343.804052] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:29:59 (1789018199) [ 2350.795187] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:30:06 (1789018206) [ 2357.778765] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:30:13 (1789018213) [ 2365.711417] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:30:20 (1789018220) [ 2374.908448] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:30:29 (1789018229) [ 2383.838975] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:30:38 (1789018238) [ 2391.968431] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:30:46 (1789018246) [ 2400.622182] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:30:55 (1789018255) [ 2408.587976] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:31:03 (1789018263) [ 2415.869529] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:31:11 (1789018271) [ 2423.587285] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:31:18 (1789018278) [ 2432.333422] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:31:27 (1789018287) [ 2442.725628] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:31:37 (1789018297) [ 2450.652421] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:31:45 (1789018305) [ 2450.748381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2450.783478] LNet: 137503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.791116] LNetError: 137503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2450.822942] LNet: Removed LNI 192.168.206.54@tcp2 [ 2450.869734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.067950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2451.134881] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2452.834649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2452.869977] LNet: 137648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.877202] LNetError: 137648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2452.904870] LNet: Removed LNI 192.168.206.54@tcp [ 2452.933532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2453.055805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2453.098656] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2461.193657] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:31:56 (1789018316) [ 2469.033394] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:32:04 (1789018324) [ 2475.455615] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:32:10 (1789018330) [ 2483.212833] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:32:18 (1789018338) [ 2490.581730] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:32:25 (1789018345) [ 2497.839842] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:32:32 (1789018352) [ 2504.803816] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:32:40 (1789018360) [ 2511.817520] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:32:47 (1789018367) [ 2518.753994] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:32:53 (1789018373) [ 2526.051888] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:33:01 (1789018381) [ 2534.428125] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:33:09 (1789018389) [ 2541.375295] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:33:16 (1789018396) [ 2548.729875] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2286 sec ========== 01:33:23 (1789018403) [ 2549.113167] LNet: 145115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2549.119405] LNetError: 145115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2549.131903] LNet: Removed LNI 192.168.206.54@tcp [ 2549.979073] Key type .llcrypt unregistered [ 2549.984037] Key type ._llcrypt unregistered [ 2562.526429] Key type ._llcrypt registered [ 2562.528960] Key type .llcrypt registered [ 2563.187341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2563.203618] alg: No test for adler32 (adler32-zlib) [ 2564.378105] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2564.615868] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2566.376267] Key type lgssc registered [ 2567.802468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2631.998992] Lustre: Mounted lustre-client - version 2.17.58_39_gf941a14 [ 2637.224499] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2645.226217] Lustre: Unmounted lustre-client [ 2671.141992] Key type lgssc unregistered [ 2671.359816] LNet: 148063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2671.370514] LNetError: 148063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2671.393591] LNet: Removed LNI 192.168.206.54@tcp [ 2671.928148] Key type .llcrypt unregistered [ 2671.933581] Key type ._llcrypt unregistered