[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 793255057 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003208] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008312] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011147] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013027] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.023113] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027601] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029096] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] *** VALIDATE tmpfs *** [ 0.031414] *** VALIDATE proc *** [ 0.033164] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043070] debug: unmapping init [mem 0xffffffff93c59000-0xffffffff93c60fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046866] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047028] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052029] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054281] rcu: Hierarchical SRCU implementation. [ 0.056419] smp: Bringing up secondary CPUs ... [ 0.058339] x86: Booting SMP configuration: [ 0.059041] .... node #0, CPUs: #1 #2 #3 [ 0.069465] smp: Brought up 1 node, 4 CPUs [ 0.071019] smpboot: Max logical packages: 1 [ 0.072014] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.116199] node 0 deferred pages initialised in 42ms [ 0.128184] devtmpfs: initialized [ 0.129463] x86/mm: Memory block size: 128MB [ 0.133745] gcov: version magic: 0x41383552 [ 0.139297] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150068] pinctrl core: initialized pinctrl subsystem [ 0.152566] [ 0.154010] ************************************************************* [ 0.158014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161013] ** ** [ 0.164015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168087] ** ** [ 0.170013] ** This means that this kernel is built to expose internal ** [ 0.173014] ** IOMMU data structures, which may compromise security on ** [ 0.176014] ** your system. ** [ 0.178010] ** ** [ 0.181015] ** If you see this message and you are not debugging the ** [ 0.184012] ** kernel, report this immediately to your vendor! ** [ 0.186012] ** ** [ 0.189014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191018] ************************************************************* [ 0.260387] NET: Registered protocol family 16 [ 0.262758] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.265058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.272166] cpuidle: using governor menu [ 0.274959] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.277680] PCI: Using configuration type 1 for base access [ 0.279201] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.288365] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.290075] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.292164] cryptd: max_cpu_qlen set to 1000 [ 0.293207] ACPI: Added _OSI(Module Device) [ 0.294095] ACPI: Added _OSI(Processor Device) [ 0.295010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.297013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.302941] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.309387] ACPI: Interpreter enabled [ 0.310054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.312014] ACPI: Using IOAPIC for interrupt routing [ 0.314090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.317391] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.329196] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.331034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.334016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.337069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.342692] acpiphp: Slot [2] registered [ 0.344122] acpiphp: Slot [5] registered [ 0.346121] acpiphp: Slot [6] registered [ 0.347119] acpiphp: Slot [3] registered [ 0.349095] acpiphp: Slot [4] registered [ 0.350079] acpiphp: Slot [7] registered [ 0.352108] acpiphp: Slot [8] registered [ 0.354542] acpiphp: Slot [9] registered [ 0.358232] acpiphp: Slot [10] registered [ 0.360112] acpiphp: Slot [11] registered [ 0.362098] acpiphp: Slot [12] registered [ 0.363135] acpiphp: Slot [13] registered [ 0.365161] acpiphp: Slot [14] registered [ 0.366087] acpiphp: Slot [15] registered [ 0.368089] acpiphp: Slot [16] registered [ 0.370104] acpiphp: Slot [17] registered [ 0.371088] acpiphp: Slot [18] registered [ 0.374116] acpiphp: Slot [19] registered [ 0.375247] acpiphp: Slot [20] registered [ 0.378000] acpiphp: Slot [21] registered [ 0.379118] acpiphp: Slot [22] registered [ 0.380000] acpiphp: Slot [23] registered [ 0.380000] acpiphp: Slot [24] registered [ 0.381087] acpiphp: Slot [25] registered [ 0.382075] acpiphp: Slot [26] registered [ 0.383102] acpiphp: Slot [27] registered [ 0.384000] acpiphp: Slot [28] registered [ 0.384000] acpiphp: Slot [29] registered [ 0.385074] acpiphp: Slot [30] registered [ 0.386100] acpiphp: Slot [31] registered [ 0.387064] PCI host bridge to bus 0000:00 [ 0.388015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.390020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.392033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.395016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.397020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.400022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.403169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.406077] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.409287] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.417011] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.421066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.424017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.425019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.428017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.430616] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.434871] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.437055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.440947] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.443916] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.454020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.458017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.462170] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.469013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.471000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.476000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.486985] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.504016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.521014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.545016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.582099] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.584676] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.587621] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.590625] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.593308] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.599356] iommu: Default domain type: Passthrough [ 0.602458] SCSI subsystem initialized [ 0.604108] ACPI: bus type USB registered [ 0.606247] usbcore: registered new interface driver usbfs [ 0.609113] usbcore: registered new interface driver hub [ 0.611214] usbcore: registered new device driver usb [ 0.614154] pps_core: LinuxPPS API ver. 1 registered [ 0.616009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.619053] PTP clock support registered [ 0.622094] EDAC MC: Ver: 3.0.0 [ 0.623000] PCI: Using ACPI for IRQ routing [ 0.625012] NetLabel: Initializing [ 0.626008] NetLabel: domain hash size = 128 [ 0.628008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.630152] NetLabel: unlabeled traffic allowed by default [ 0.633463] vgaarb: loaded [ 0.635420] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.638014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.648535] clocksource: Switched to clocksource kvm-clock [ 1.363853] VFS: Disk quotas dquot_6.6.0 [ 1.365901] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.368873] *** VALIDATE ramfs *** [ 1.370301] *** VALIDATE hugetlbfs *** [ 1.372849] pnp: PnP ACPI init [ 1.375345] pnp: PnP ACPI: found 6 devices [ 1.392289] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.395747] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.398158] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.400384] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.402734] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.405472] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.408354] NET: Registered protocol family 2 [ 1.410811] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.415692] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.419635] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.425657] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.429405] TCP: Hash tables configured (established 65536 bind 65536) [ 1.432575] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.435900] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.438436] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.441450] NET: Registered protocol family 1 [ 1.443768] RPC: Registered named UNIX socket transport module. [ 1.445954] RPC: Registered udp transport module. [ 1.456411] RPC: Registered tcp transport module. [ 1.458641] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.466426] NET: Registered protocol family 44 [ 1.468268] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.470726] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.473078] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.476050] PCI: CLS 0 bytes, default 64 [ 1.478121] Unpacking initramfs... [ 4.512775] debug: unmapping init [mem 0xffff992f7cc64000-0xffff992f7ffcffff] [ 4.517644] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.520380] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.523656] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 6.134635] Initialise system trusted keyrings [ 6.136333] Key type blacklist registered [ 6.139460] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.151968] zbud: loaded [ 6.156377] *** VALIDATE nfs *** [ 6.161108] *** VALIDATE nfs4 *** [ 6.165644] pstore: using deflate compression [ 6.173506] Platform Keyring initialized [ 6.375515] NET: Registered protocol family 38 [ 6.389986] Key type asymmetric registered [ 6.393407] Asymmetric key parser 'x509' registered [ 6.396869] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.403314] io scheduler mq-deadline registered [ 6.406693] io scheduler kyber registered [ 6.409562] io scheduler bfq registered [ 6.414253] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.427821] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.432814] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.453404] ACPI: Power Button [PWRF] [ 6.575560] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.587220] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.747264] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.776393] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.807341] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.813626] Non-volatile memory driver v1.3 [ 6.814907] Linux agpgart interface v0.103 [ 6.846796] virtio_blk virtio1: [vda] 134856 512-byte logical blocks (69.0 MB/65.8 MiB) [ 6.849328] vda: detected capacity change from 0 to 69046272 [ 6.889151] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.892492] vdb: detected capacity change from 0 to 1073741824 [ 6.899933] libphy: Fixed MDIO Bus: probed [ 6.906213] usbcore: registered new interface driver usbserial_generic [ 6.971786] usbserial: USB Serial support registered for generic [ 6.983844] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.024792] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.041087] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.067875] mousedev: PS/2 mouse device common for all mice [ 7.109648] rtc_cmos 00:05: RTC can wake from S4 [ 7.124549] rtc_cmos 00:05: registered as rtc0 [ 7.128646] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.136452] intel_pstate: CPU model not supported [ 7.138321] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.149315] hid: raw HID events driver (C) Jiri Kosina [ 7.151345] usbcore: registered new interface driver usbhid [ 7.153585] usbhid: USB HID core driver [ 7.155196] drop_monitor: Initializing network drop monitor service [ 7.157673] Initializing XFRM netlink socket [ 7.159940] NET: Registered protocol family 10 [ 7.164176] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.170560] Segment Routing with IPv6 [ 7.177196] NET: Registered protocol family 17 [ 7.181648] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.195917] mpls_gso: MPLS GSO support [ 7.215266] RAS: Correctable Errors collector initialized. [ 7.228142] AVX version of gcm_enc/dec engaged. [ 7.231368] AES CTR mode by8 optimization enabled [ 7.324442] sched_clock: Marking stable (7324422996, 0)->(9121324989, -1796901993) [ 7.328486] registered taskstats version 1 [ 7.330596] Loading compiled-in X.509 certificates [ 7.333738] zswap: loaded using pool lzo/zbud [ 7.369401] Key type big_key registered [ 7.386510] Key type encrypted registered [ 7.388642] ima: No TPM chip found, activating TPM-bypass! [ 7.391401] ima: Allocated hash algorithm: sha1 [ 7.393450] ima: No architecture policies found [ 7.394989] evm: Initialising EVM extended attributes: [ 7.397399] evm: security.selinux [ 7.398802] evm: security.ima [ 7.400233] evm: security.capability [ 7.401616] evm: HMAC attrs: 0x1 [ 7.404387] rtc_cmos 00:05: setting system clock to 2026-06-01 16:16:09 UTC (1780330569) [ 7.412655] debug: unmapping init [mem 0xffffffff94c03000-0xffffffff94dfffff] [ 7.419716] debug: unmapping init [mem 0xffffffff93982000-0xffffffff93c58fff] [ 7.426341] Write protecting the kernel read-only data: 28672k [ 7.430424] debug: unmapping init [mem 0xffffffff92003000-0xffffffff921fffff] [ 7.434529] debug: unmapping init [mem 0xffffffff92914000-0xffffffff929fffff] [ 7.494519] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.514855] systemd[1]: Detected virtualization kvm. [ 7.517271] systemd[1]: Detected architecture x86-64. [ 7.522642] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.575721] systemd[1]: No hostname configured. [ 7.578746] systemd[1]: Set hostname to . [ 7.582601] random: systemd: uninitialized urandom read (16 bytes read) [ 7.589311] systemd[1]: Initializing machine ID from random generator. [ 7.738792] random: ln: uninitialized urandom read (6 bytes read) [ 7.827955] random: systemd: uninitialized urandom read (16 bytes read) [ 7.835202] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.852828] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 7.862602] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.372521] device-mapper: uevent: version 1.0.3 [ 11.376951] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 15.916493] random: fast init done [ 16.376494] virtio_net virtio0 ens2: renamed from eth0 [ 18.042487] scsi host0: ata_piix [ 18.202516] scsi host1: ata_piix [ 18.217559] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 18.242535] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 24.164778] random: crng init done [ 24.179225] random: 7 urandom warning(s) missed due to ratelimiting [ 25.672514] hrtimer: interrupt took 4582076 ns [ 27.902558] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 31.770021] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 43.179499] printk: systemd: 26 output lines suppressed due to ratelimiting [ 44.930973] SELinux: Disabled at runtime. [ 45.020693] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 45.055218] systemd[1]: Detected virtualization kvm. [ 45.063398] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 48.770262] systemd[1]: initrd-switch-root.service: Succeeded. [ 48.784928] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 48.802510] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 48.810410] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 48.823964] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 48.843831] systemd[1]: Starting Journal Service... Starting Journal Service... [ 48.859358] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pa[ 49.164239] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ges File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ 50.124411] systemd[1]: sysroot-var-lib-nfs-rpc_pipefs.mount: Succeeded. [ 50.145756] systemd[1]: sysroot.mount: Succeeded. [ 50.170779] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 51.622093] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 52.917964] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 53.206368] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 53.670758] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 54.042037] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 61.058657] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 62.473301] NFS: Registering the id_resolver key type [ 62.483099] Key type id_resolver registered [ 62.493347] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg457-client login: [ 161.953303] libcfs: loading out-of-tree module taints kernel. [ 162.240963] Key type ._llcrypt registered [ 162.243027] Key type .llcrypt registered [ 162.944607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 162.969981] alg: No test for adler32 (adler32-zlib) [ 164.505760] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 165.708582] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 167.536243] Key type lgssc registered [ 169.446098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 301.714202] Lustre: Mounted lustre-client [ 306.921908] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 326.471497] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing check_logdir /tmp/testlogs/ [ 327.136259] Lustre: lustre-OST0000-osc-ffff992fcb0b5800: disconnect after 23s idle [ 332.462727] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing yml_node [ 337.819847] Lustre: DEBUG MARKER: Client: 2.17.53.30 [ 340.437637] Lustre: DEBUG MARKER: MDS: 2.17.53.30 [ 343.035643] Lustre: DEBUG MARKER: OSS: 2.17.53.30 [ 345.083568] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jun 1 12:21:45 EDT 2026 [ 363.636975] LustreError: 7964:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff992fcb0b5800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 363.690330] Lustre: Unmounted lustre-client [ 385.537674] Key type lgssc unregistered [ 385.959487] LNet: 8409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.982655] LNetError: 8409:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 385.996752] LNet: Removed LNI 192.168.204.57@tcp [ 387.018853] Key type .llcrypt unregistered [ 387.023205] Key type ._llcrypt unregistered [ 417.530479] Key type ._llcrypt registered [ 417.533492] Key type .llcrypt registered [ 417.840937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 417.864041] alg: No test for adler32 (adler32-zlib) [ 418.766518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 418.839459] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 418.895918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 423.765201] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 423.946709] LNet: 9289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.956462] LNetError: 9289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 423.999566] LNet: Removed LNI 192.168.204.57@tcp [ 424.819558] Key type .llcrypt unregistered [ 424.822979] Key type ._llcrypt unregistered [ 432.296838] Key type ._llcrypt registered [ 432.299092] Key type .llcrypt registered [ 433.873648] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:23:14 (1780330994) [ 434.388281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 434.419767] alg: No test for adler32 (adler32-zlib) [ 435.428318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.588490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.614159] LNet: 9706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.645841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.738397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.758676] LNet: 9713:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 444.229255] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:23:24 (1780331004) [ 444.347738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.393664] LNet: 10285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.444105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.559366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 452.012748] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:23:32 (1780331012) [ 452.102408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.139442] LNet: 10872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.189158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.333310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 459.767448] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:23:40 (1780331020) [ 460.087072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.128356] LNet: 11462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.155577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.286298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 468.099448] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:23:48 (1780331028) [ 468.290494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.330569] LNet: 12046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.353157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.457620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 469.088708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 477.087980] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:23:57 (1780331037) [ 477.283148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.301318] LNet: 12637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 477.320875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.431372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 478.108827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 478.735676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 487.075478] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:24:07 (1780331047) [ 487.200291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.256595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.371608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 495.044416] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:24:15 (1780331055) [ 495.384209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.439246] LNet: 13819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 495.452825] LNet: 13819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 495.483905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.621228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 495.701030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 496.295172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 496.383563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 497.262959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 497.303568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 498.001755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 498.053718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 498.426735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 498.513684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 499.281259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 499.316686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 500.157460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 500.206694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 500.830982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 500.883101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 501.626725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 501.676203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 502.271333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 502.303676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 502.849146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 502.919172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 503.501844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 503.544956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 504.010909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 504.099585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 512.796315] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:24:33 (1780331073) [ 512.912657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.997965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.161210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 513.224197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 520.911275] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:24:41 (1780331081) [ 521.094106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.158835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.265628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 521.328760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 529.534811] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:24:49 (1780331089) [ 529.645691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.711400] LNet: 15668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.719663] LNet: 15668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 529.765413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.883902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 529.921331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 538.884869] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:24:58 (1780331098) [ 539.048172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.177652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.353554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 539.442708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 548.932406] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:25:08 (1780331108) [ 549.060062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.150123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.305855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 549.381123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 557.734707] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:25:17 (1780331117) [ 557.863533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.959060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.080073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 558.125192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 565.553395] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:25:26 (1780331126) [ 565.649948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.709883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.821649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 566.114980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 573.837890] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:25:34 (1780331134) [ 574.188743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.259350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.419635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 574.686030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 582.507658] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:25:43 (1780331143) [ 582.658309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.736075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.877945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 583.187363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 590.149170] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:25:50 (1780331150) [ 590.430558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.484325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.965750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 591.431708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 599.950698] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:26:00 (1780331160) [ 600.304463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.338397] LNet: 20505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.349240] LNet: 20505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 600.373744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.453502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 600.546377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 609.544760] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:26:09 (1780331169) [ 609.870878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.937394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.065966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 610.168790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 618.914558] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:26:19 (1780331179) [ 619.176779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.220396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.313394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 627.048488] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:26:27 (1780331187) [ 627.160145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.201642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.307118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 635.266307] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:26:35 (1780331195) [ 635.494114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.522231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.635677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 636.371978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 644.885893] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:26:45 (1780331205) [ 645.006736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.045485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.127409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 645.166395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 653.805474] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:26:53 (1780331213) [ 653.936971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.030560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.162926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 654.210480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 661.999325] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:27:02 (1780331222) [ 662.230801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.291409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.424550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 662.482489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 669.232953] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:27:09 (1780331229) [ 669.368408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.442426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.585451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 669.627443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 676.390713] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:27:17 (1780331237) [ 676.710645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.782061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.872882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 676.969873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 677.015413] LNetError: 25825:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 677.090160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 677.145243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 684.084153] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:27:24 (1780331244) [ 684.323703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.393875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.525557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q0ScYhRtec [ 684.605343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q0ScYhRtec [ 684.699962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q0ScYhRtec [ 692.023199] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:27:32 (1780331252) [ 692.278382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.337088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.429585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 692.494083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 692.548708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 700.306518] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:27:40 (1780331260) [ 700.482600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.538825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.657722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 703.744363] LNet: Message forwarding will be enabled in 2s [ 706.848216] LNet: Message forwarding enabled [ 707.351387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 707.392609] LNet: Message forwarding will stop in 2s to 6s [ 710.994416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 711.838140] LNet: Message forwarding disabled [ 714.143833] LNet: Message forwarding will be enabled in 2s [ 717.985093] LNet: Message forwarding enabled [ 721.399708] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:28:02 (1780331282) [ 721.639340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.276593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.373372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 729.802983] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:28:10 (1780331290) [ 729.913976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.942592] LNet: 28839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.948467] LNet: 28839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 729.975627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.037621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 730.062935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 730.090298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 730.130343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 730.166280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 730.205867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 730.244209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 730.283369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 730.314650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 730.342702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 730.373640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 730.399622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 730.437335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 730.468255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 730.508713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 730.548741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 730.583239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 730.616478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 730.655342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 730.677497] LNetError: 28888:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 737.576545] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:28:18 (1780331298) [ 737.672781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.748643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.862584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 745.348591] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:28:25 (1780331305) [ 747.590314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg457-server [ 750.109469] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:28:30 (1780331310) [ 750.264647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.319481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.386798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.426286] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 752.242697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[58-62]@tcp [ 752.280325] LNet: 30606:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 752.348158] LNetError: 30395:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.58@tcp is being used as a gateway but routing feature is not turned on [ 752.376315] LNetError: 30395:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.58@tcp has gone from up to down [ 753.397833] LNetError: 30395:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.58@tcp is being used as a gateway but routing feature is not turned on [ 753.421928] LNetError: 30395:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 754.412567] LNetError: 30395:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.58@tcp is being used as a gateway but routing feature is not turned on [ 754.432605] LNetError: 30395:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 755.426064] LNetError: 30607:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.61@tcp has gone from up to down [ 755.441819] LNetError: 30607:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 756.531329] LNetError: 30395:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.59@tcp is being used as a gateway but routing feature is not turned on [ 756.551684] LNetError: 30395:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 760.624747] LNetError: 30395:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.58@tcp is being used as a gateway but routing feature is not turned on [ 760.658940] LNetError: 30395:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 11 previous similar messages [ 760.942311] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:28:41 (1780331321) [ 763.044556] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg457-server [ 764.707408] LNetError: 30394:0:(router.c:1262:lnet_check_routers()) Failed to discover router 192.168.204.61@tcp [ 764.961202] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:28:45 (1780331325) [ 765.046165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.071271] LNetError: 31535:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.096524] LNet: Removed LNI 192.168.204.57@tcp [ 765.123687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.213888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 765.230868] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 765.944501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[58-63/2]@tcp [ 765.966368] LNet: 31660:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 765.972217] LNet: 31660:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 766.004843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[58-63/2]@tcp [ 766.006875] LNetError: 31545:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.58@tcp has gone from up to down [ 766.037346] LNetError: 31545:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 773.252401] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:28:53 (1780331333) [ 773.355066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.400443] LNetError: 32244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.417470] LNet: Removed LNI 192.168.204.57@tcp [ 773.492143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.596429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 773.646483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.676281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.748975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 773.806691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.834899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.903074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 773.982364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.013334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.072798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 774.149982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.194179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.238679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 774.306915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.352601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.428651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 774.505648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.562172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.637359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 774.730384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.785251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.884936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 775.012782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.046571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.097372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 775.188142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.236222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.327654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 775.426596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.476926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.534039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 775.608927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.639603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.707964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 782.717155] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:29:03 (1780331343) [ 784.387484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg457-server [ 786.775177] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:29:07 (1780331347) [ 789.251089] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg457-server [ 791.540676] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:29:11 (1780331351) [ 793.124199] Key type .llcrypt unregistered [ 793.127036] Key type ._llcrypt unregistered [ 799.889665] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 800.071105] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 800.073412] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 800.087722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 802.836335] Key type ._llcrypt registered [ 802.846068] Key type .llcrypt registered [ 803.045947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.064883] alg: No test for adler32 (adler32-zlib) [ 804.037847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.129974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.182651] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 804.990643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 805.020886] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 807.740189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 807.794943] LNet: Removed LNI 10.1.2.3@tcp [ 808.083615] LNet: 34376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.096635] LNetError: 34376:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.151208] LNet: Removed LNI 192.168.204.57@tcp [ 809.285405] Key type .llcrypt unregistered [ 809.290778] Key type ._llcrypt unregistered [ 821.296554] Key type ._llcrypt registered [ 821.300543] Key type .llcrypt registered [ 822.905896] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:29:43 (1780331383) [ 823.635294] Key type .llcrypt unregistered [ 823.641832] Key type ._llcrypt unregistered [ 830.739830] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 830.874701] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 830.877944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 830.897516] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 832.513618] Key type ._llcrypt registered [ 832.519332] Key type .llcrypt registered [ 832.766635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.788716] alg: No test for adler32 (adler32-zlib) [ 833.725838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.810372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.856413] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 834.775745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 834.813256] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 838.829721] LNet: 35857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.836400] LNetError: 35857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.840112] LNet: Removed LNI 192.168.204.57@tcp [ 838.863311] LNet: Removed LNI 10.1.2.3@tcp [ 839.813192] Key type .llcrypt unregistered [ 839.818572] Key type ._llcrypt unregistered [ 851.763776] Key type ._llcrypt registered [ 851.766623] Key type .llcrypt registered [ 853.543627] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:30:13 (1780331413) [ 854.266445] Key type .llcrypt unregistered [ 854.277352] Key type ._llcrypt unregistered [ 860.575392] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 860.723691] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 860.728691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 860.732541] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 863.539807] Key type ._llcrypt registered [ 863.541658] Key type .llcrypt registered [ 863.759323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.795056] alg: No test for adler32 (adler32-zlib) [ 864.726422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.788041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 864.828551] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 867.285790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 867.321221] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 869.806666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 869.833357] LNet: Removed LNI 10.1.2.3@tcp [ 869.855417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 869.895581] LNet: Removed LNI 10.1.2.31@tcp [ 869.924696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 869.964617] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 870.777469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 870.793605] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 871.884892] LNet: 37778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.904505] LNetError: 37778:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 871.922151] LNet: Removed LNI 10.1.2.3@tcp [ 871.946712] LNet: Removed LNI 10.1.2.31@tcp [ 872.920322] Key type .llcrypt unregistered [ 872.926861] Key type ._llcrypt unregistered [ 888.042068] Key type ._llcrypt registered [ 888.050426] Key type .llcrypt registered [ 889.879895] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:30:49 (1780331449) [ 890.427186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 890.471876] alg: No test for adler32 (adler32-zlib) [ 891.354251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.431478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 891.461227] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 892.109785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.126633] LNet: 38732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.131100] LNetError: 38732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.143801] LNet: Removed LNI 192.168.204.57@tcp [ 892.154530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.191718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.57@tcp [ 892.225657] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 893.128209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.57@tcp [ 893.156387] LNet: Removed LNI 192.168.204.57@tcp [ 893.213830] LNet: 38867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.952511] Key type .llcrypt unregistered [ 893.956892] Key type ._llcrypt unregistered [ 907.079542] Key type ._llcrypt registered [ 907.087238] Key type .llcrypt registered [ 908.728794] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:31:08 (1780331468) [ 911.504674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg457-server [ 913.769274] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:31:13 (1780331473) [ 914.749166] Key type .llcrypt unregistered [ 914.758426] Key type ._llcrypt unregistered [ 922.075988] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 922.197579] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 922.220710] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 922.233269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 924.976576] Key type ._llcrypt registered [ 924.979308] Key type .llcrypt registered [ 925.223892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.251394] alg: No test for adler32 (adler32-zlib) [ 926.199545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.262913] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 927.357740] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 929.706192] LNet: 40708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.710783] LNetError: 40708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.722796] LNet: Removed LNI 192.168.204.57@tcp1 [ 929.731937] LNet: Removed LNI 10.1.2.3@tcp2 [ 930.574495] Key type .llcrypt unregistered [ 930.578076] Key type ._llcrypt unregistered [ 941.423673] Key type ._llcrypt registered [ 941.425716] Key type .llcrypt registered [ 942.635757] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:31:43 (1780331503) [ 943.092426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.113642] alg: No test for adler32 (adler32-zlib) [ 944.100677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.158904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 944.182497] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 945.166414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.200357] LNet: 41677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.211208] LNetError: 41677:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.226259] LNet: Removed LNI 192.168.204.57@tcp [ 945.246294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.330681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 945.355846] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 951.664972] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:31:52 (1780331512) [ 951.817418] LNet: 42266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.830339] LNetError: 42266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.854417] LNet: Removed LNI 192.168.204.57@tcp [ 952.327528] Key type .llcrypt unregistered [ 952.329215] Key type ._llcrypt unregistered [ 957.802963] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 957.935435] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 957.947634] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 957.957333] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 960.592928] Key type ._llcrypt registered [ 960.596321] Key type .llcrypt registered [ 960.776684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.791424] alg: No test for adler32 (adler32-zlib) [ 961.720751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.783865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.814460] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 962.661603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 962.688856] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.326558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.344566] LNet: 43012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.350284] LNetError: 43012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.355607] LNet: Removed LNI 192.168.204.57@tcp [ 965.367818] LNet: Removed LNI 10.1.2.3@tcp [ 965.393921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.470762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 965.501184] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 965.507179] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 966.156771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.174858] LNet: 43039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.181900] LNetError: 43039:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.192872] LNetError: 43039:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 966.198925] LNet: Removed LNI 192.168.204.57@tcp [ 966.206554] LNet: Removed LNI 10.1.2.3@tcp [ 966.224418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.276644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 966.303664] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 966.309949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 966.967484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 966.992142] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 967.903886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.921746] LNet: 43201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.927531] LNetError: 43201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.931222] LNetError: 43201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 967.936144] LNet: Removed LNI 192.168.204.57@tcp [ 967.937865] LNet: Removed LNI 10.1.2.3@tcp [ 967.946608] LNet: Removed LNI 192.168.204.57@tcp2 [ 967.967182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.029254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 968.064194] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 968.071531] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 968.077130] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 976.314046] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:32:16 (1780331536) [ 976.444801] LNet: 43792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.455276] LNetError: 43792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.462092] LNetError: 43792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 976.473196] LNet: Removed LNI 192.168.204.57@tcp [ 976.477691] LNet: Removed LNI 10.1.2.3@tcp [ 976.493458] LNet: Removed LNI 192.168.204.57@tcp2 [ 977.234713] Key type .llcrypt unregistered [ 977.237231] Key type ._llcrypt unregistered [ 984.304360] Key type ._llcrypt registered [ 984.309435] Key type .llcrypt registered [ 984.549239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.576960] alg: No test for adler32 (adler32-zlib) [ 985.527087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.582291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 985.615437] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 986.508221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 986.545683] LNet: 44194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.555881] LNetError: 44194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.579605] LNet: Removed LNI 192.168.204.57@tcp [ 986.609768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.675873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 986.696098] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 987.189469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.209494] LNet: 44225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.218253] LNetError: 44225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.239181] LNet: Removed LNI 192.168.204.57@tcp [ 987.270322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.363949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 987.410339] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 987.926334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.961971] LNet: Removed LNI 192.168.204.57@tcp [ 987.985991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.068807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 988.104931] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 988.653762] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 988.876195] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 988.902878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 988.911936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 991.337916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.356273] LNet: 44334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.364853] LNet: 44334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 991.370647] LNetError: 44334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.375602] LNetError: 44334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 991.388189] LNet: Removed LNI 192.168.204.57@tcp [ 991.404910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.465676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 991.484445] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 998.974972] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:32:39 (1780331559) [ 999.074875] LNet: 44924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.080778] LNetError: 44924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.103723] LNet: Removed LNI 192.168.204.57@tcp [ 999.729133] Key type .llcrypt unregistered [ 999.731764] Key type ._llcrypt unregistered [ 1006.078581] Key type ._llcrypt registered [ 1006.080706] Key type .llcrypt registered [ 1006.275431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1006.297153] alg: No test for adler32 (adler32-zlib) [ 1007.242640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.297889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1007.329382] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1008.208650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.224554] LNet: 45330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.232569] LNetError: 45330:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.242897] LNet: Removed LNI 192.168.204.57@tcp [ 1008.264992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.366085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1008.412837] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1015.263116] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:32:55 (1780331575) [ 1015.569075] LNet: 45922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.578216] LNetError: 45922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.592607] LNet: Removed LNI 192.168.204.57@tcp [ 1016.290177] Key type .llcrypt unregistered [ 1016.298201] Key type ._llcrypt unregistered [ 1023.762129] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1023.929104] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1023.938463] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1023.947410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1026.713238] Key type ._llcrypt registered [ 1026.715513] Key type .llcrypt registered [ 1026.853566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1026.865787] alg: No test for adler32 (adler32-zlib) [ 1027.857961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.934426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1027.999097] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1028.973409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1029.000738] LNet: 46373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.009742] LNetError: 46373:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.029916] LNet: Removed LNI 192.168.204.57@tcp [ 1029.053938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.142204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1029.166359] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1036.601709] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:33:17 (1780331597) [ 1036.804173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.853989] LNet: 46970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.862688] LNetError: 46970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.878426] LNet: Removed LNI 192.168.204.57@tcp [ 1036.901266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.012616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1043.601617] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:33:24 (1780331604) [ 1043.740260] LNet: 47544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.492909] Key type .llcrypt unregistered [ 1044.496196] Key type ._llcrypt unregistered [ 1050.447173] Key type ._llcrypt registered [ 1050.449409] Key type .llcrypt registered [ 1050.717327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.756582] alg: No test for adler32 (adler32-zlib) [ 1051.746593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.815208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1051.849665] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1052.518690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1052.537691] LNet: 47950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.545847] LNetError: 47950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.566070] LNet: Removed LNI 192.168.204.57@tcp [ 1052.608511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.663667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1052.692677] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1058.894743] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:33:39 (1780331619) [ 1059.039881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1059.067549] LNet: 48538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.075077] LNetError: 48538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.110497] LNet: Removed LNI 192.168.204.57@tcp [ 1059.151212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.224722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1064.841200] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:33:45 (1780331625) [ 1066.086844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1067.436798] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:33:48 (1780331628) [ 1069.030337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1070.890756] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:33:51 (1780331631) [ 1072.627605] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1074.589804] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:33:55 (1780331635) [ 1076.600788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1078.474779] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:33:59 (1780331639) [ 1080.150790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1081.953762] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:34:02 (1780331642) [ 1082.054953] LNet: 50587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.610805] Key type .llcrypt unregistered [ 1082.614640] Key type ._llcrypt unregistered [ 1088.529941] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1088.683678] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1088.686128] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1088.702213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1091.396759] Key type ._llcrypt registered [ 1091.398448] Key type .llcrypt registered [ 1091.589861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.605457] alg: No test for adler32 (adler32-zlib) [ 1092.488990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.553807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1092.570439] LNet: Added LNI 192.168.204.57@tcp [8/512/0/180] [ 1092.576539] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1092.579141] LNet: 50916:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1092.698985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1092.720835] LNet: 50935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.731233] LNetError: 50935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1092.739785] LNet: Removed LNI 192.168.204.57@tcp [ 1092.742415] LNetError: 50935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1092.769561] LNet: Removed LNI 10.1.2.3@tcp [ 1092.816102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.887856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1092.912250] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1092.982780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1092.998949] LNet: 50959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.005234] LNetError: 50959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1093.022710] LNet: Removed LNI 192.168.204.57@tcp [ 1093.049424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.146972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1093.174976] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1093.181335] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1093.185768] LNet: 50968:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1093.276922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1093.303710] LNet: 50986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.313774] LNetError: 50986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1093.324619] LNet: Removed LNI 192.168.204.57@tcp [ 1093.326806] LNetError: 50986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1093.338238] LNet: Removed LNI 10.1.2.3@tcp [ 1093.367685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.431621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1093.455220] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1093.461570] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1093.463866] LNet: 50994:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1093.645360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1093.669653] LNet: 51020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.677118] LNetError: 51020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1093.686363] LNet: Removed LNI 192.168.204.57@tcp [ 1093.690930] LNetError: 51020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1093.711848] LNet: Removed LNI 10.1.2.3@tcp [ 1093.752959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.812263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1093.835815] LNet: Added LNI 192.168.204.57@tcp [8/256/0/182] [ 1093.840108] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1093.927389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1093.948417] LNet: 51048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.954211] LNetError: 51048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1093.957357] LNet: Removed LNI 192.168.204.57@tcp [ 1093.961653] LNetError: 51048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1093.981538] LNet: Removed LNI 10.1.2.3@tcp [ 1094.008752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.067316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1094.085330] LNet: Added LNI 192.168.204.57@tcp [8/256/0/182] [ 1094.092939] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1094.095901] LNet: 51056:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1094.159595] LNet: Removed LNI 192.168.204.57@tcp [ 1094.176385] LNet: Removed LNI 10.1.2.3@tcp [ 1094.706996] Key type .llcrypt unregistered [ 1094.711409] Key type ._llcrypt unregistered [ 1105.286207] Key type ._llcrypt registered [ 1105.288806] Key type .llcrypt registered [ 1106.628530] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:34:27 (1780331667) [ 1107.090917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1107.104873] alg: No test for adler32 (adler32-zlib) [ 1107.987156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1108.036895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1108.063551] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1108.721864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.57 [ 1108.737262] LNet: 52031:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1108.744336] LNet: Removed LNI 192.168.204.57@tcp [ 1116.683360] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:34:37 (1780331677) [ 1116.809821] LNet: 52592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.827102] LNetError: 52592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1116.853958] LNet: Removed LNI 192.168.204.57@tcp [ 1117.576537] Key type .llcrypt unregistered [ 1117.583596] Key type ._llcrypt unregistered [ 1125.304082] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1125.554853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1128.228406] Key type ._llcrypt registered [ 1128.230413] Key type .llcrypt registered [ 1128.370646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1128.388587] alg: No test for adler32 (adler32-zlib) [ 1129.827951] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1137.403426] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:34:57 (1780331697) [ 1137.558504] LNet: 53507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.569529] LNetError: 53507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1137.619378] LNet: Removed LNI 10.1.2.3@tcp [ 1138.479756] Key type .llcrypt unregistered [ 1138.496490] Key type ._llcrypt unregistered [ 1144.166459] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1144.444364] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1144.454493] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1144.471597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1147.299639] Key type ._llcrypt registered [ 1147.305522] Key type .llcrypt registered [ 1147.530468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1147.558887] alg: No test for adler32 (adler32-zlib) [ 1148.527298] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1155.401398] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:35:16 (1780331716) [ 1157.178419] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1158.935568] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:35:19 (1780331719) [ 1159.009576] LNet: 54691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.020684] LNetError: 54691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1159.041444] LNet: Removed LNI 10.1.2.3@tcp [ 1159.725125] Key type .llcrypt unregistered [ 1159.729334] Key type ._llcrypt unregistered [ 1165.517543] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1165.728798] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1165.739226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1165.744557] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1168.467312] Key type ._llcrypt registered [ 1168.473262] Key type .llcrypt registered [ 1168.678178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1168.704032] alg: No test for adler32 (adler32-zlib) [ 1169.653762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1169.758433] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1177.678635] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:35:38 (1780331738) [ 1178.198242] LNet: 55929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.221507] LNetError: 55929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1178.249282] LNet: Removed LNI 10.1.2.3@tcp [ 1178.987424] Key type .llcrypt unregistered [ 1178.990950] Key type ._llcrypt unregistered [ 1184.832576] Key type ._llcrypt registered [ 1184.835764] Key type .llcrypt registered [ 1185.064470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1185.089864] alg: No test for adler32 (adler32-zlib) [ 1185.979422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1185.999539] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1192.007383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1192.561400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_interrupt -r 1 [ 1192.594637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_interrupt -r 1 [ 1192.634250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1192.649293] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1192.818252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.844755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1192.871945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.897728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.924111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.957623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.992678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.030207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.059864] LNet: 56367:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.081397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.115905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.142567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.165667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.186847] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1193.190552] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1193.194957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.217124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.260162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1193.725574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_dropped -r 1 [ 1193.752605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_dropped -r 1 [ 1193.800839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1194.029365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.064787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1194.099893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1194.125802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1194.170162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1194.206050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1194.256992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1194.302981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1194.318639] LNet: 56476:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1194.334867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.364747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.416541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.453885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.491684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.538864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.581627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1195.087235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_aborted -r 1 [ 1195.136772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_aborted -r 1 [ 1195.172126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1195.188935] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1195.194989] LNet: Skipped 1 previous similar message [ 1195.406648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1195.448082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1195.483652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.534583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.559757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.582549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.619137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.645511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.664952] LNet: 56585:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1195.683273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.721266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.751525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.786988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.820740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.850809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.882250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1196.393521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_no_route -r 1 [ 1196.435548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_no_route -r 1 [ 1196.475868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1196.716640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.754103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1196.788591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.821519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.856943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.896703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.939940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.977798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.029577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.071211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.118498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.145619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.173788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.204729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.242959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.748985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_timeout -r 1 [ 1197.781646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_timeout -r 1 [ 1197.812648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1197.838676] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1197.850977] LNet: Skipped 2 previous similar messages [ 1198.163746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1198.199616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1198.238529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.266593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.296744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.333114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.361660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.398864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.444801] LNet: 56803:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1198.452469] LNet: 56803:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1198.477960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.526692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1198.559588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.586589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.616484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.647610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.675489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1199.280280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_error -r 1 [ 1199.312739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_error -r 1 [ 1199.372390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1199.665804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.701903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1199.751202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.772720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.800529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.849289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.900151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.939576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.981952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1200.009080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1200.042948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.099619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.145400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.185775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.229068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1200.501033] LNet: 56954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.512464] LNetError: 56954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.533171] LNet: Removed LNI 192.168.204.57@tcp [ 1200.999461] Key type .llcrypt unregistered [ 1201.001258] Key type ._llcrypt unregistered [ 1219.579788] Key type ._llcrypt registered [ 1219.590188] Key type .llcrypt registered [ 1221.005716] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:36:21 (1780331781) [ 1221.784339] Key type .llcrypt unregistered [ 1221.786990] Key type ._llcrypt unregistered [ 1227.853615] Key type ._llcrypt registered [ 1227.859605] Key type .llcrypt registered [ 1228.157944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1228.173791] alg: No test for adler32 (adler32-zlib) [ 1229.074432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1229.105080] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1236.392373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1245.793225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1245.835444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1245.859563] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1247.082393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_interrupt -r 1 [ 1247.109683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_interrupt -r 1 [ 1247.145831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_interrupt -r 1 [ 1247.172380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_interrupt -r 1 [ 1247.202179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_interrupt -r 1 [ 1247.232156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_interrupt -r 1 [ 1247.263109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_interrupt -r 1 [ 1247.290421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_interrupt -r 1 [ 1247.315840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1247.335960] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1247.342240] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1247.540814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.566550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1247.598301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.629714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.656983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.700391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.754274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.825356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.847580] LNet: 58420:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.885294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.934387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.969227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.017741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.040054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.067236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.097569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1248.509384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_dropped -r 1 [ 1248.565242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_dropped -r 1 [ 1248.633278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_dropped -r 1 [ 1248.677453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_dropped -r 1 [ 1248.720491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_dropped -r 1 [ 1248.759475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_dropped -r 1 [ 1248.816611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_dropped -r 1 [ 1248.870147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_dropped -r 1 [ 1248.914744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1248.942991] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1248.949857] LNet: Skipped 2 previous similar messages [ 1249.200629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.240581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1249.273727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.300529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.335209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.372985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.421738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.464531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.490992] LNet: 58541:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.508820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.558473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.608203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.639350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.661359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.704907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.738387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1250.197900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_aborted -r 1 [ 1250.231958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_aborted -r 1 [ 1250.278071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_aborted -r 1 [ 1250.337747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_aborted -r 1 [ 1250.372349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_aborted -r 1 [ 1250.401991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_aborted -r 1 [ 1250.427556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_aborted -r 1 [ 1250.457548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_aborted -r 1 [ 1250.493779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1250.509410] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1250.513810] LNet: Skipped 2 previous similar messages [ 1250.713874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1250.747562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1250.780272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1250.812241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1250.869919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1250.908776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1250.947995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1250.976796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1250.997286] LNet: 58662:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1251.013867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.043240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.076327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.113336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.163424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.200131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.237688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1251.775559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_no_route -r 1 [ 1251.829015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_no_route -r 1 [ 1251.871532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_no_route -r 1 [ 1251.909852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_no_route -r 1 [ 1251.962549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_no_route -r 1 [ 1252.020604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_no_route -r 1 [ 1252.072577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_no_route -r 1 [ 1252.119060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_no_route -r 1 [ 1252.160824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1252.383248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.422855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1252.453694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1252.476891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1252.507085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.535973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.565469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.592335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.619833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.656864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.681475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.716437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.750064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1252.802568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.834169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1253.358920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_timeout -r 1 [ 1253.399441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_timeout -r 1 [ 1253.446698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_timeout -r 1 [ 1253.497333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_timeout -r 1 [ 1253.539764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_timeout -r 1 [ 1253.597470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_timeout -r 1 [ 1253.646132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_timeout -r 1 [ 1253.682629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_timeout -r 1 [ 1253.710832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1253.729919] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1253.737073] LNet: Skipped 5 previous similar messages [ 1253.950721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1253.981602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1254.010398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.038655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.061786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.085629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.108942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.133988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.161214] LNet: 58905:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1254.169619] LNet: 58905:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1254.182750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1254.210745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.241921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.286438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1254.316781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.344942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1254.370571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1254.887588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_error -r 1 [ 1254.934561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_error -r 1 [ 1254.978838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_error -r 1 [ 1255.020480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_error -r 1 [ 1255.073653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_error -r 1 [ 1255.130508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_error -r 1 [ 1255.171271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_error -r 1 [ 1255.195273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_error -r 1 [ 1255.252748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1255.498418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1255.533838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1255.577680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1255.627428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1255.658679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1255.693874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1255.722364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1255.759130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1255.800261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.823329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.852773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.887488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.915258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.942917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.970913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1261.148849] LNet: 59102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.157059] LNetError: 59102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.169497] LNet: Removed LNI 192.168.204.57@tcp [ 1261.188255] LNet: Removed LNI 192.168.204.57@tcp1 [ 1261.614603] Key type .llcrypt unregistered [ 1261.617319] Key type ._llcrypt unregistered [ 1276.453175] Key type ._llcrypt registered [ 1276.456882] Key type .llcrypt registered [ 1277.812542] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:37:18 (1780331838) [ 1279.049917] Key type .llcrypt unregistered [ 1279.055587] Key type ._llcrypt unregistered [ 1284.415195] Key type ._llcrypt registered [ 1284.419865] Key type .llcrypt registered [ 1284.561980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1284.575693] alg: No test for adler32 (adler32-zlib) [ 1285.464280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1285.504318] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1291.716821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1292.385379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_dropped -r 1 [ 1292.450896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_dropped -r 1 [ 1292.503473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1292.544399] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1292.783130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.816828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1292.858521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.899689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.939217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.989813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1293.034445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1293.079818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1293.113322] LNet: 60351:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1293.142595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1293.202181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1293.260768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.309510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.385950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.452613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.493377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1294.293066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_error -r 1 [ 1294.334660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_error -r 1 [ 1294.392839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1294.410688] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1294.629500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1294.668511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1294.716242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1294.748448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1294.784557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1294.809462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1294.835334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1294.879696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1294.900126] LNet: 60460:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1294.916577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1294.944694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1294.980540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1295.020771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1295.067976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1295.097458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1295.137147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1295.709631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_timeout -r 1 [ 1295.759265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_timeout -r 1 [ 1295.796202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1295.845946] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1296.117657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1296.157739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1296.198281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1296.244449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1296.302211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1296.386044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1296.442357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1296.520590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1296.580693] LNet: 60569:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1296.613100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1296.659415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.706701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.772804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.820948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.867933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.910763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1297.303875] LNet: 60611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.314330] LNetError: 60611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.334117] LNet: Removed LNI 192.168.204.57@tcp [ 1297.825694] Key type .llcrypt unregistered [ 1297.828958] Key type ._llcrypt unregistered [ 1314.219255] Key type ._llcrypt registered [ 1314.223093] Key type .llcrypt registered [ 1315.748322] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:37:56 (1780331876) [ 1316.532994] Key type .llcrypt unregistered [ 1316.539572] Key type ._llcrypt unregistered [ 1321.993534] Key type ._llcrypt registered [ 1321.999573] Key type .llcrypt registered [ 1322.266946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1322.284405] alg: No test for adler32 (adler32-zlib) [ 1323.245654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1323.299975] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1329.350400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1337.861750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.929480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1337.957609] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1339.736938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_dropped -r 1 [ 1339.788570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e remote_dropped -r 1 [ 1339.845341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_dropped -r 1 [ 1339.913734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e remote_dropped -r 1 [ 1339.990912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e remote_dropped -r 1 [ 1340.066036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e remote_dropped -r 1 [ 1340.133734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e remote_dropped -r 1 [ 1340.208479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e remote_dropped -r 1 [ 1340.249939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1340.290605] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1340.303109] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.157@tcp [ 1340.559959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1340.600612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1340.659446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1340.695330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1340.723419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1340.774403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1340.822830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1340.866603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1340.892684] LNet: 62076:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1340.921630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1340.975636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.013822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.086123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.127184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.163168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.199517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1341.257461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.858574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_error -r 1 [ 1341.900336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e remote_error -r 1 [ 1341.967292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_error -r 1 [ 1342.019435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e remote_error -r 1 [ 1342.086446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e remote_error -r 1 [ 1342.141752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e remote_error -r 1 [ 1342.194160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e remote_error -r 1 [ 1342.234166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e remote_error -r 1 [ 1342.284830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1342.308093] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1342.312648] LNet: Skipped 2 previous similar messages [ 1342.588808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.633794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1342.661138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.696173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1342.734912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1342.768698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1342.809853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1342.841725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1342.869820] LNet: 62199:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1342.894816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1342.956709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1342.993631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.058191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.099293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.171165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.214515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1343.249608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.761563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_timeout -r 1 [ 1343.807469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e remote_timeout -r 1 [ 1343.873959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_timeout -r 1 [ 1343.925583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e remote_timeout -r 1 [ 1343.981190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e remote_timeout -r 1 [ 1344.043758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e remote_timeout -r 1 [ 1344.084838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e remote_timeout -r 1 [ 1344.140800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e remote_timeout -r 1 [ 1344.211758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.157@tcp [ 1344.230345] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1344.436974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1344.464740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1344.495480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1344.536559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1344.590463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1344.623219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1344.673794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1344.731982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1344.765568] LNet: 62323:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1344.805296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1344.851046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1344.885525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1344.920708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1344.980161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1345.021580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1345.048754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1345.092625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1350.297435] LNet: 62401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1350.306527] LNetError: 62401:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1350.315249] LNet: Removed LNI 192.168.204.57@tcp [ 1350.346261] LNet: Removed LNI 192.168.204.57@tcp1 [ 1350.809204] Key type .llcrypt unregistered [ 1350.828930] Key type ._llcrypt unregistered [ 1366.504507] Key type ._llcrypt registered [ 1366.507022] Key type .llcrypt registered [ 1367.587790] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:38:48 (1780331928) [ 1369.317120] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1370.885933] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:38:51 (1780331931) [ 1371.735693] Key type .llcrypt unregistered [ 1371.740606] Key type ._llcrypt unregistered [ 1377.745955] Key type ._llcrypt registered [ 1377.752259] Key type .llcrypt registered [ 1378.008832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1378.041522] alg: No test for adler32 (adler32-zlib) [ 1379.004504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1379.064182] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1386.745923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1387.733980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e network_timeout -r 1 [ 1387.781216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e network_timeout -r 1 [ 1387.828447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1387.859040] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1388.110956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1388.141568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1388.173727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1388.202279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1388.234620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1388.271769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1388.310641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1388.322106] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1388.371194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1388.407052] LNet: 63943:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1388.443706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1388.489976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1388.546947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1388.605932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1388.643569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1388.693446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1389.044381] LNet: 63982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.051432] LNetError: 63982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.070462] LNet: Removed LNI 192.168.204.57@tcp [ 1389.501454] Key type .llcrypt unregistered [ 1389.503189] Key type ._llcrypt unregistered [ 1407.030929] Key type ._llcrypt registered [ 1407.036132] Key type .llcrypt registered [ 1407.340544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1407.360927] alg: No test for adler32 (adler32-zlib) [ 1408.281525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1408.304924] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1415.102456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1423.672427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.709478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1423.735669] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1425.283823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e network_timeout -r 1 [ 1425.332229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e network_timeout -r 1 [ 1425.366953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e network_timeout -r 1 [ 1425.428945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e network_timeout -r 1 [ 1425.483916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e network_timeout -r 1 [ 1425.539286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e network_timeout -r 1 [ 1425.593862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e network_timeout -r 1 [ 1425.641960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e network_timeout -r 1 [ 1425.709777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1425.737543] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1426.017356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1426.065497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1426.121939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1426.191610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1426.261833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1426.315460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1426.374129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1426.454793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1426.483478] LNet: 64861:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1426.504043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1426.544640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1426.579820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1426.638201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1426.679176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1426.717056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1426.722327] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1426.743467] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1428.770641] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1431.070294] LNet: 64935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.077679] LNetError: 64935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.096168] LNet: Removed LNI 192.168.204.57@tcp [ 1432.100104] LNetError: 64935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.120117] LNet: Removed LNI 192.168.204.57@tcp1 [ 1432.594556] Key type .llcrypt unregistered [ 1432.596501] Key type ._llcrypt unregistered [ 1447.935410] Key type ._llcrypt registered [ 1447.941181] Key type .llcrypt registered [ 1449.042617] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:40:09 (1780332009) [ 1449.541520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1449.567385] alg: No test for adler32 (adler32-zlib) [ 1450.471338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.519836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.542488] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1451.283683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1451.326099] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1452.149578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 1452.234879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1452.291815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1452.337869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1452.382535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1452.417135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1452.452659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.57@tcp [ 1452.477981] LNetError: 66035:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.57@tcp to recovery [ 1452.576383] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1452.578056] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1454.624605] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1454.877142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1458.720763] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1459.500485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1459.536213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1459.588311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1459.604292] LNet: 66142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.609732] LNetError: 66142:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.641063] LNet: Removed LNI 192.168.204.57@tcp [ 1460.654645] LNetError: 66142:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.687276] LNet: Removed LNI 192.168.204.57@tcp1 [ 1460.730449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.854850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1460.896035] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1461.691682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1461.715731] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1462.851129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 1462.895425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1463.000933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1463.066649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1463.112642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1463.168070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1463.238730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1463.271954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.57@tcp [ 1463.304593] LNetError: 66415:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.57@tcp to recovery [ 1463.840806] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1466.950751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1469.985387] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1469.993122] LNet: Skipped 1 previous similar message [ 1474.075621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1474.123338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1474.174986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1480.653841] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:40:41 (1780332041) [ 1480.750942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.788886] LNet: 67116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.826158] LNetError: 67116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.832518] LNet: Removed LNI 192.168.204.57@tcp [ 1480.847959] LNet: Removed LNI 192.168.204.57@tcp1 [ 1480.907591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.950432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.972828] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1481.699843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.713318] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1482.481717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 1482.566655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1482.610799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1482.655499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1482.699733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1482.744888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1482.809214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.57@tcp --health 0 [ 1482.908801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1482.978169] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.57@tcp [ 1484.000527] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1484.009062] LNet: Skipped 1 previous similar message [ 1497.395882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1497.562405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1497.598196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1497.636958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.57@tcp --health 500 [ 1497.748503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1504.405088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1504.472366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1504.499383] LNet: 67607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.507492] LNetError: 67607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.512118] LNetError: 67607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1504.517888] LNet: Removed LNI 192.168.204.57@tcp [ 1504.528153] LNet: Removed LNI 192.168.204.57@tcp1 [ 1504.557112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.636237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1504.666634] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1505.835847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1505.866797] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1506.736172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 1506.782236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1506.862485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1506.886389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1506.927827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1506.971197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1507.000819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1507.034475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.57@tcp --health 0 [ 1508.704655] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1508.708081] LNet: Skipped 3 previous similar messages [ 1509.414249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.632685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.690822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1519.727948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1528.025559] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:41:28 (1780332088) [ 1528.654241] LNet: 68664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.664871] LNetError: 68664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.671342] LNetError: 68664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1528.683579] LNet: Removed LNI 192.168.204.57@tcp [ 1528.702575] LNet: Removed LNI 192.168.204.57@tcp1 [ 1529.714142] Key type .llcrypt unregistered [ 1529.719575] Key type ._llcrypt unregistered [ 1536.337400] Key type ._llcrypt registered [ 1536.342409] Key type .llcrypt registered [ 1536.619867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1536.633414] alg: No test for adler32 (adler32-zlib) [ 1537.553584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1537.607946] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1546.877898] Lustre: DEBUG MARKER: Initial discovery [ 1547.062885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.157@tcp [ 1548.687327] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1548.824697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1548.863844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.157@tcp [ 1548.884755] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1548.896356] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1548.920114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1548.950730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.157@tcp -r 1 -m GET -l 3 [ 1549.001531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.157@tcp -r 1 -m GET -e local_error [ 1549.040543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.157@tcp -r 1 -m PUT -l 6 [ 1550.161336] Lustre: DEBUG MARKER: Force oleg457-client.virtnet to discover 192.168.204.157@tcp (in background) [ 1550.229580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.157@tcp [ 1551.538322] Lustre: DEBUG MARKER: Force oleg457-server to discover 192.168.204.57@tcp [ 1553.319334] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1553.489707] Lustre: DEBUG MARKER: Wait for 69420 [ 1554.670702] Lustre: DEBUG MARKER: Finished wait on 69420 [ 1554.769646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1554.800136] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1554.824253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1555.132616] LNet: 69877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.142556] LNetError: 69877:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.163853] LNet: Removed LNI 192.168.204.57@tcp [ 1555.576844] Key type .llcrypt unregistered [ 1555.580808] Key type ._llcrypt unregistered [ 1569.308802] Key type ._llcrypt registered [ 1569.311431] Key type .llcrypt registered [ 1570.517823] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:42:11 (1780332131) [ 1571.055432] Key type .llcrypt unregistered [ 1571.057369] Key type ._llcrypt unregistered [ 1576.907827] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1577.051531] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1577.059839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1577.066126] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1579.642884] Key type ._llcrypt registered [ 1579.645789] Key type .llcrypt registered [ 1579.851301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1579.869451] alg: No test for adler32 (adler32-zlib) [ 1580.789532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.874521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1580.904859] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1581.580738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1581.608348] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1589.390128] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:42:30 (1780332150) [ 1589.596435] LNet: 71954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.616815] LNetError: 71954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.633649] LNet: Removed LNI 192.168.204.57@tcp [ 1589.651564] LNet: Removed LNI 10.1.2.3@tcp [ 1590.299711] Key type .llcrypt unregistered [ 1590.306479] Key type ._llcrypt unregistered [ 1596.089799] Key type ._llcrypt registered [ 1596.093716] Key type .llcrypt registered [ 1596.268273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1596.281538] alg: No test for adler32 (adler32-zlib) [ 1597.201242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.275516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1597.302929] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1597.910374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1597.934795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1598.763195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.57@tcp [ 1599.840239] LNetError: Unexpected error -22 connecting to 192.168.204.57@tcp at host 192.168.204.57:988 [ 1604.659808] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:42:45 (1780332165) [ 1604.893435] LNet: 73087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.904157] LNetError: 73087:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.920383] LNet: Removed LNI 192.168.204.57@tcp [ 1605.922589] LNetError: 73087:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1605.958436] LNet: Removed LNI 10.1.2.3@tcp [ 1606.542715] Key type .llcrypt unregistered [ 1606.545443] Key type ._llcrypt unregistered [ 1611.655340] Key type ._llcrypt registered [ 1611.657921] Key type .llcrypt registered [ 1611.813320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1611.825393] alg: No test for adler32 (adler32-zlib) [ 1612.727288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.789273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1612.819231] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1613.466573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1613.481781] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 1614.329364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.57@tcp1 --nid 192.168.204.57@tcp2 [ 1618.166541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp1 192.168.204.57@tcp2 [ 1618.194755] LNetError: 73923:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.57@tcp1 to 192.168.204.57@tcp2 - Route table may be misconfigured [ 1618.221187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp2 192.168.204.57@tcp1 [ 1624.097776] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:43:04 (1780332184) [ 1624.382507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.417959] LNet: 74496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.427660] LNetError: 74496:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.443356] LNet: Removed LNI 192.168.204.57@tcp1 [ 1625.519589] LNet: Removed LNI 192.168.204.57@tcp2 [ 1625.570818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.667183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1625.695367] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1626.318751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1626.346719] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1627.379687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 1627.423853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.57@tcp [ 1627.456814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp1 192.168.204.57@tcp1 [ 1627.493942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.57@tcp -d 192.168.204.57@tcp -e network_timeout [ 1627.528821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -e network_timeout [ 1627.576618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -e network_timeout [ 1627.612121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -e network_timeout [ 1627.663616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1627.686690] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1627.706985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1627.795668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1627.902297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1628.704440] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1634.044179] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:43:14 (1780332194) [ 1634.162282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.183096] LNet: 75356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.194093] LNetError: 75356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.198953] LNetError: 75356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1634.212209] LNet: Removed LNI 192.168.204.57@tcp [ 1635.241529] LNet: Removed LNI 192.168.204.57@tcp1 [ 1635.268587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.363390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1635.640845] LNet: 75395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.019164] Key type .llcrypt unregistered [ 1636.021595] Key type ._llcrypt unregistered [ 1646.032385] Key type ._llcrypt registered [ 1646.035448] Key type .llcrypt registered [ 1647.110096] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:43:27 (1780332207) [ 1647.630896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1647.650734] alg: No test for adler32 (adler32-zlib) [ 1648.601992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.730183] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1651.366421] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:43:32 (1780332212) [ 1651.482774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.507370] LNet: 76489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1651.533830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.613240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1651.632168] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1652.125689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1652.142564] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1652.868836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1652.896092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp1 [ 1652.929130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp1 [ 1659.094182] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:43:39 (1780332219) [ 1661.233662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg457-server [ 1663.354405] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:43:43 (1780332223) [ 1665.223564] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg457-server [ 1667.310968] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:43:47 (1780332227) [ 1668.815650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg457-server [ 1670.848378] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:43:51 (1780332231) [ 1672.207919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg457-server [ 1674.309431] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:43:55 (1780332235) [ 1676.095598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg457-server [ 1677.482594] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:43:58 (1780332238) [ 1679.179841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg457-server [ 1680.458463] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:44:01 (1780332241) [ 1682.111484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg457-server [ 1683.609382] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:44:04 (1780332244) [ 1685.131776] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg457-server [ 1686.635321] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:44:07 (1780332247) [ 1688.140117] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg457-server [ 1689.579848] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:44:10 (1780332250) [ 1689.674176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.691496] LNet: 80367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.696635] LNetError: 80367:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.709241] LNet: Removed LNI 192.168.204.57@tcp [ 1690.731498] LNet: Removed LNI 192.168.204.57@tcp1 [ 1690.768155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.835671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1690.868568] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1691.633158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1691.709039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1691.913167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.942943] LNet: 80519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.952275] LNetError: 80519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.962397] LNetError: 80519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1691.977251] LNet: Removed LNI 192.168.204.57@tcp [ 1691.994753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.048931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1692.079859] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1692.734887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1692.968328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.987943] LNetError: 80664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.058344] LNet: Removed LNI 192.168.204.57@tcp [ 1694.110361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.178634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.218599] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1694.888935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1694.954281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1695.172727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.206556] LNet: 80811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.216337] LNet: 80811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1695.227969] LNetError: 80811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.252277] LNet: Removed LNI 192.168.204.57@tcp [ 1695.290935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.389822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1695.441959] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1696.580178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1698.117806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.188605] LNet: Removed LNI 192.168.204.57@tcp [ 1699.209982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.265918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1699.288288] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1700.062611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1700.224595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1700.668830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.714469] LNet: 81113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.722684] LNet: 81113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1700.749276] LNetError: 81113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.773458] LNetError: 81113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1700.813830] LNet: Removed LNI 192.168.204.57@tcp [ 1700.869371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.956587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1701.008896] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1702.290356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1702.527954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.594366] LNet: Removed LNI 192.168.204.57@tcp [ 1703.675950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.788034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.837717] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1704.635808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1704.702551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1704.985936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.030427] LNet: Removed LNI 192.168.204.57@tcp [ 1706.103166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.179977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1706.247955] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1707.100500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1707.372867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.464467] LNet: Removed LNI 192.168.204.57@tcp [ 1708.506111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.603470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.655585] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1709.705096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1709.845063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1710.121037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.146415] LNet: 81699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.153605] LNet: 81699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1710.162771] LNetError: 81699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.177758] LNetError: 81699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1710.197326] LNet: Removed LNI 192.168.204.57@tcp [ 1710.229695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.340414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1710.390191] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1711.459350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1711.759455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.809632] LNet: Removed LNI 192.168.204.57@tcp [ 1712.832882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.899908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.930964] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1713.781929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1713.873950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1714.157269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.251108] LNet: Removed LNI 192.168.204.57@tcp [ 1715.314549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.486386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1715.546928] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1716.825874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1718.413339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.468756] LNet: Removed LNI 192.168.204.57@tcp [ 1719.513589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.600715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1719.638345] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1720.484369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1720.551194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.609413] LNet: Removed LNI 192.168.204.57@tcp [ 1720.645302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.718893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1720.767971] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1721.956264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1729.944080] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:44:50 (1780332290) [ 1730.085124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.140028] LNet: 82977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.146697] LNet: 82977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1730.161891] LNetError: 82977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.167156] LNetError: 82977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1730.189127] LNet: Removed LNI 192.168.204.57@tcp [ 1730.226578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.289208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1730.315141] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/180] [ 1731.602061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.646674] LNet: Removed LNI 192.168.204.57@tcp231 [ 1731.679537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.803367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1731.844394] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/180] [ 1731.854294] LNet: 83129:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1732.388500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1732.423672] LNet: Removed LNI 192.168.204.57@tcp231 [ 1732.438815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1732.490639] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/180] [ 1733.492429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.523819] LNet: Removed LNI 192.168.204.57@tcp231 [ 1733.552800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.678059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1733.721925] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/0] [ 1733.727977] LNet: 83281:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1734.147605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1734.207267] LNet: Removed LNI 192.168.204.57@tcp231 [ 1734.225092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1734.273473] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/0] [ 1735.688800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.722754] LNet: Removed LNI 192.168.204.57@tcp231 [ 1735.749874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.834768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1735.868982] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/60] [ 1735.882919] LNet: 83433:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1736.258798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1736.310903] LNet: Removed LNI 192.168.204.57@tcp231 [ 1736.337176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1736.379098] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/60] [ 1737.868282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.953249] LNet: Removed LNI 192.168.204.57@tcp231 [ 1737.974167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.051522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1738.082244] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/180] [ 1738.090049] LNet: 83584:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1738.570971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1738.609604] LNet: Removed LNI 192.168.204.57@tcp231 [ 1738.644700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1738.674738] LNet: Added LNI 192.168.204.57@tcp231 [8/256/0/180] [ 1745.573808] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:45:06 (1780332306) [ 1745.915805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.954803] LNet: Removed LNI 192.168.204.57@tcp231 [ 1745.999625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.077595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1746.112870] LNet: Added LNI 192.168.204.57@tcp232 [8/256/0/180] [ 1746.714447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1746.743543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1746.773175] LNet: Removed LNI 192.168.204.57@tcp232 [ 1746.804542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1746.849213] LNet: Added LNI 192.168.204.57@tcp232 [8/256/0/180] [ 1747.628630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1747.670826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1747.699357] LNet: Removed LNI 192.168.204.57@tcp232 [ 1747.722187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1747.757830] LNet: Added LNI 192.168.204.57@tcp232 [8/256/0/180] [ 1748.550219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1748.616874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1748.680384] LNet: Removed LNI 192.168.204.57@tcp232 [ 1748.692838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1748.719546] LNet: Added LNI 192.168.204.57@tcp232 [8/256/0/180] [ 1749.535566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1749.583725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1749.622390] LNet: Removed LNI 192.168.204.57@tcp232 [ 1749.643191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1756.118706] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:45:16 (1780332316) [ 1757.233157] Key type .llcrypt unregistered [ 1757.238277] Key type ._llcrypt unregistered [ 1762.935661] Key type ._llcrypt registered [ 1762.937298] Key type .llcrypt registered [ 1763.126383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1763.158452] alg: No test for adler32 (adler32-zlib) [ 1764.250859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1764.291694] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1772.175914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1780.951518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.025798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1781.063839] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1782.557574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_interrupt -r 2 [ 1782.606844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_interrupt -r 2 [ 1782.638082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_interrupt -r 2 [ 1782.679405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_interrupt -r 2 [ 1782.726644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_interrupt -r 2 [ 1782.769197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_interrupt -r 2 [ 1782.804424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_interrupt -r 2 [ 1782.858842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_interrupt -r 2 [ 1782.916640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1782.950467] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1782.959506] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1783.233417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1783.284169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1783.312673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1783.340386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1783.386401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1783.442645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1783.502823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1783.544968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1783.571479] LNet: 86084:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1783.584884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1783.610689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1783.643507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1783.685411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1783.727874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1783.759077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1784.269508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_dropped -r 2 [ 1784.305493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_dropped -r 2 [ 1784.329497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_dropped -r 2 [ 1784.360661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_dropped -r 2 [ 1784.387674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_dropped -r 2 [ 1784.422378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_dropped -r 2 [ 1784.458625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_dropped -r 2 [ 1784.510938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_dropped -r 2 [ 1784.567848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1784.602497] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1784.866429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1784.913328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1784.968699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1785.012669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1785.071313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1785.115594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1785.160381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1785.192548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1785.220435] LNet: 86204:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1785.245469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1785.283563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1785.319533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1785.382158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1785.429967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1785.477688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1786.092255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_aborted -r 2 [ 1786.140839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_aborted -r 2 [ 1786.183870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_aborted -r 2 [ 1786.224867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_aborted -r 2 [ 1786.273224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_aborted -r 2 [ 1786.313687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_aborted -r 2 [ 1786.358246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_aborted -r 2 [ 1786.410724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_aborted -r 2 [ 1786.474669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1786.513798] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1786.765926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1786.790260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1786.814359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1786.836630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1786.878828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1786.930437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1786.966619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1787.009828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1787.040123] LNet: 86324:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1787.068594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1787.106459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1787.135658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1787.162565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1787.192354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1787.225386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1787.893901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_no_route -r 2 [ 1787.928296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_no_route -r 2 [ 1787.979278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_no_route -r 2 [ 1788.020528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_no_route -r 2 [ 1788.067604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_no_route -r 2 [ 1788.125969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_no_route -r 2 [ 1788.183252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_no_route -r 2 [ 1788.248953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_no_route -r 2 [ 1788.291581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1788.536557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1788.581308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1788.615987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1788.662618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1788.704903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1788.735959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1788.798911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1788.858991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1788.904765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1788.937485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1788.980228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1789.014319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1789.043383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1789.078243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1789.775890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e local_timeout -r 2 [ 1789.838996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e local_timeout -r 2 [ 1789.882856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e local_timeout -r 2 [ 1789.935142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e local_timeout -r 2 [ 1789.970517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e local_timeout -r 2 [ 1790.022648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e local_timeout -r 2 [ 1790.069555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e local_timeout -r 2 [ 1790.153565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e local_timeout -r 2 [ 1790.189071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1790.206872] LNet: There was an unexpected network error while writing to 192.168.204.157: rc = -22 [ 1790.214345] LNet: Skipped 1 previous similar message [ 1790.525977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1790.596076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1790.633551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1790.661936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1790.690949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1790.725483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1790.762345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1790.834412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1790.856873] LNet: 86565:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1790.863731] LNet: 86565:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1790.894178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1790.937298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1790.984114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1791.015597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1791.048561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1791.085857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1791.735761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp -m GET -e remote_dropped -r 2 [ 1791.774422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.157@tcp1 -m GET -e remote_dropped -r 2 [ 1791.811927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp -m GET -e remote_dropped -r 2 [ 1791.854317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp -d 192.168.204.57@tcp1 -m GET -e remote_dropped -r 2 [ 1791.882558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp -m GET -e remote_dropped -r 2 [ 1791.921458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.157@tcp1 -m GET -e remote_dropped -r 2 [ 1791.963334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp -m GET -e remote_dropped -r 2 [ 1791.995952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.57@tcp1 -d 192.168.204.57@tcp1 -m GET -e remote_dropped -r 2 [ 1792.058501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.57@tcp 192.168.204.157@tcp [ 1792.091519] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.157@tcp [ 1792.347829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1792.378710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1792.406938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1792.430779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1792.471800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1792.497600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1792.558176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1792.600345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1792.628463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1792.669526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1792.712246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1792.750179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1792.820943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1792.866380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1798.734383] LNet: 86759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.745802] LNetError: 86759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.753812] LNet: Removed LNI 192.168.204.57@tcp [ 1798.775154] LNet: Removed LNI 192.168.204.57@tcp1 [ 1799.141500] Key type .llcrypt unregistered [ 1799.143716] Key type ._llcrypt unregistered [ 1813.666414] Key type ._llcrypt registered [ 1813.677962] Key type .llcrypt registered [ 1815.046493] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:46:15 (1780332375) [ 1815.541246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1815.567607] alg: No test for adler32 (adler32-zlib) [ 1816.486243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.557532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1816.583498] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1817.396470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1817.438929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.57@tcp [ 1817.475514] LNetError: 87737:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.57@tcp to recovery [ 1817.568785] LNet: 1 local NIs in recovery (showing 1): 192.168.204.57@tcp [ 1817.604732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.57@tcp [ 1817.716706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1824.918199] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:46:25 (1780332385) [ 1825.126992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1825.148820] LNet: 88327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.154825] LNetError: 88327:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.172943] LNet: Removed LNI 192.168.204.57@tcp [ 1825.203976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.294473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1825.325857] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1826.009843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1826.031538] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 1826.862357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.57@tcp [ 1826.876476] LNetError: 88593:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.57@tcp to recovery [ 1827.298667] LNet: There was an unexpected network error while writing to 192.168.204.57: rc = -22 [ 1828.174689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.57@tcp [ 1836.375853] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:46:36 (1780332396) [ 1838.243666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg457-server [ 1840.452911] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:46:40 (1780332400) [ 1842.367724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg457-server [ 1844.129935] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:46:44 (1780332404) [ 1845.862515] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg457-server [ 1847.265776] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:46:48 (1780332408) [ 1847.472473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1847.501542] LNet: 90224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.509842] LNetError: 90224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.520591] LNet: Removed LNI 192.168.204.57@tcp [ 1847.537620] LNet: Removed LNI 192.168.204.57@tcp1 [ 1847.580611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.623906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1847.665767] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1848.264903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1848.325444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1848.359575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1848.422884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1848.459732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1848.501357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1854.270735] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:46:55 (1780332415) [ 1854.417960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.442665] LNet: 90949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.448595] LNetError: 90949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.457408] LNetError: 90949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1854.478749] LNet: Removed LNI 192.168.204.57@tcp [ 1854.497372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.560447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1854.584678] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1861.518603] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:47:02 (1780332422) [ 1861.663415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1861.687547] LNet: 91650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1861.698701] LNetError: 91650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1861.744762] LNet: Removed LNI 192.168.204.57@tcp [ 1861.772499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.844383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1861.868784] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1869.260802] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:47:09 (1780332429) [ 1870.855819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1872.860304] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:47:13 (1780332433) [ 1873.288342] LNet: 92686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.299865] LNetError: 92686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.310493] LNet: Removed LNI 192.168.204.57@tcp [ 1873.948794] Key type .llcrypt unregistered [ 1873.952761] Key type ._llcrypt unregistered [ 1880.848069] Key type ._llcrypt registered [ 1880.850226] Key type .llcrypt registered [ 1881.254972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1881.280394] alg: No test for adler32 (adler32-zlib) [ 1882.278411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1882.346848] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1892.003071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 1898.225971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.157@tcp [ 1898.435358] LNet: 93104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.441452] LNetError: 93104:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.463222] LNet: Removed LNI 192.168.204.57@tcp [ 1898.761556] Key type .llcrypt unregistered [ 1898.768215] Key type ._llcrypt unregistered [ 1908.586294] Key type ._llcrypt registered [ 1908.588624] Key type .llcrypt registered [ 1910.182309] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1911.665821] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1913.822187] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:47:54 (1780332474) [ 1915.446310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1917.691647] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:47:58 (1780332478) [ 1919.522282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg457-server [ 1921.327477] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:48:02 (1780332482) [ 1923.297291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg457-server [ 1924.869243] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:48:05 (1780332485) [ 1926.277492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1926.311606] alg: No test for adler32 (adler32-zlib) [ 1927.368421] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 1927.544060] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1929.177872] Key type lgssc registered [ 1930.372224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1950.690057] Key type lgssc unregistered [ 1950.939591] LNet: 96276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.948273] LNetError: 96276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1950.966940] LNet: Removed LNI 192.168.204.57@tcp [ 1951.535584] Key type .llcrypt unregistered [ 1951.537495] Key type ._llcrypt unregistered [ 1965.205491] Key type ._llcrypt registered [ 1965.207725] Key type .llcrypt registered [ 1966.433024] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:48:47 (1780332527) [ 1966.942888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1966.953662] alg: No test for adler32 (adler32-zlib) [ 1967.872679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.939105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1967.969980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1968.014671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1968.066931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1974.501257] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:48:55 (1780332535) [ 1974.784668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.816714] LNet: 97691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.834276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.877143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1974.905267] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1975.752869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.776367] LNet: 97835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.781765] LNetError: 97835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.794274] LNet: Removed LNI 192.168.204.57@tcp [ 1975.810812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.850391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1975.868298] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1976.360510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.388413] LNetError: 97863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1976.409278] LNet: Removed LNI 192.168.204.57@tcp [ 1976.433254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.512301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1976.542546] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1982.981758] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:49:03 (1780332543) [ 1983.209840] LNet: 98454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.216792] LNet: 98454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1983.226645] LNetError: 98454:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.252114] LNet: Removed LNI 192.168.204.57@tcp [ 1983.880380] Key type .llcrypt unregistered [ 1983.888071] Key type ._llcrypt unregistered [ 1990.042068] Key type ._llcrypt registered [ 1990.044756] Key type .llcrypt registered [ 1990.215488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1990.238468] alg: No test for adler32 (adler32-zlib) [ 1991.204716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.331783] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 1991.337060] LNet: *** cfs_fail_loc=e003, val=100*** [ 1992.091404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 1992.116496] LNet: *** cfs_fail_loc=e003, val=100*** [ 1992.125681] LNet: Skipped 2 previous similar messages [ 1992.142494] Ping reply received with mapping: [ 1992.145546] found metadata reply NI[192.168.204.57@tcp] [ 1992.148282] 39 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1992.155378] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1992.159858] 5a 5a 5a 5a 5a 5a 5a 5a [ 1992.372040] LNet: 98852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.387996] LNetError: 98852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.413056] LNet: Removed LNI 192.168.204.57@tcp [ 1993.157434] Key type .llcrypt unregistered [ 1993.160346] Key type ._llcrypt unregistered [ 2002.515630] Key type ._llcrypt registered [ 2002.518771] Key type .llcrypt registered [ 2003.869329] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:49:24 (1780332564) [ 2004.595487] Key type .llcrypt unregistered [ 2004.597953] Key type ._llcrypt unregistered [ 2010.558236] Key type ._llcrypt registered [ 2010.561253] Key type .llcrypt registered [ 2010.726979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2010.741300] alg: No test for adler32 (adler32-zlib) [ 2011.871533] LNetError: 99898:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2012.285385] Key type .llcrypt unregistered [ 2012.289440] Key type ._llcrypt unregistered [ 2017.372617] Key type ._llcrypt registered [ 2017.376427] Key type .llcrypt registered [ 2018.403137] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:49:39 (1780332579) [ 2018.936648] Key type .llcrypt unregistered [ 2018.941074] Key type ._llcrypt unregistered [ 2024.214091] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2024.218050] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2026.590920] Key type ._llcrypt registered [ 2026.593288] Key type .llcrypt registered [ 2026.737647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2026.751593] alg: No test for adler32 (adler32-zlib) [ 2027.635727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.685706] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2035.588443] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:49:56 (1780332596) [ 2035.672248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.697253] LNet: 101669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.702324] LNetError: 101669:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2035.719191] LNet: Removed LNI 10.1.2.3@tcp [ 2035.745402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.310939] Key type .llcrypt unregistered [ 2036.314050] Key type ._llcrypt unregistered [ 2041.330869] Key type ._llcrypt registered [ 2041.336869] Key type .llcrypt registered [ 2041.483149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2041.496181] alg: No test for adler32 (adler32-zlib) [ 2042.393126] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2042.444994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 2042.590304] LNet: 101965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.604884] LNetError: 101965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.620265] LNet: Removed LNI 192.168.204.57@tcp [ 2043.196803] Key type .llcrypt unregistered [ 2043.199473] Key type ._llcrypt unregistered [ 2054.145570] Key type ._llcrypt registered [ 2054.148250] Key type .llcrypt registered [ 2055.204955] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:50:16 (1780332616) [ 2055.676464] Key type .llcrypt unregistered [ 2055.680269] Key type ._llcrypt unregistered [ 2061.191465] Key type ._llcrypt registered [ 2061.193671] Key type .llcrypt registered [ 2061.408737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2061.424494] alg: No test for adler32 (adler32-zlib) [ 2068.934813] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:50:29 (1780332629) [ 2069.215539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.291131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.424308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2069.473880] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2070.355507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2070.388778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2070.421973] LNet: Removed LNI 192.168.204.57@tcp [ 2070.683784] LNet: 103708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.006460] Key type .llcrypt unregistered [ 2071.010127] Key type ._llcrypt unregistered [ 2085.298255] Key type ._llcrypt registered [ 2085.300611] Key type .llcrypt registered [ 2086.468960] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:50:47 (1780332647) [ 2087.240600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2087.263964] alg: No test for adler32 (adler32-zlib) [ 2088.285827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.363955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2088.398654] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2089.387144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 2089.454956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.57@tcp [ 2089.478468] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2089.485629] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2089.496580] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2089, 0, 2139 [ 2089.510962] LNet: Conn 0 sched [nconns, cpt]: 2, 0 [ 2089.523887] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2089.532465] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2089.539111] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2089, 0, 2139 [ 2089.551906] LNet: Conn 1 sched [nconns, cpt]: 2, 0 [ 2096.801266] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:50:57 (1780332657) [ 2097.596833] LNet: 105248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.607214] LNetError: 105248:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2097.626745] LNet: Removed LNI 192.168.204.57@tcp [ 2097.992515] Key type .llcrypt unregistered [ 2097.996520] Key type ._llcrypt unregistered [ 2103.572047] Key type ._llcrypt registered [ 2103.574191] Key type .llcrypt registered [ 2103.776748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2103.796641] alg: No test for adler32 (adler32-zlib) [ 2104.822987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2104.854629] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2112.637215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 2122.364109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.419669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2122.446591] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 2123.772221] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2123.971157] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2123.979506] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2123.989562] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2126.469573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2126.504826] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2136.175376] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:51:37 (1780332697) [ 2136.711516] LNet: 106756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.717453] LNetError: 106756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.725235] LNet: Removed LNI 192.168.204.57@tcp [ 2137.760366] LNet: Removed LNI 192.168.204.57@tcp1 [ 2137.764228] LNetError: 106756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2137.771177] LNetError: 106756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2137.784847] LNet: Removed LNI 10.1.2.3@tcp99 [ 2138.416782] Key type .llcrypt unregistered [ 2138.420221] Key type ._llcrypt unregistered [ 2144.685813] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2144.693812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2147.098974] Key type ._llcrypt registered [ 2147.100966] Key type .llcrypt registered [ 2147.229873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2147.241608] alg: No test for adler32 (adler32-zlib) [ 2148.196716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.252512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2148.287589] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2148.882439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2148.913233] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2150.854789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2150.904078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2150.948101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2150.968287] LNetError: 107501:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2151.004463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2151.046237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2151.449853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2151.532394] LNet: 107521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.538064] LNetError: 107521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2151.542525] LNet: Removed LNI 192.168.204.57@tcp [ 2151.556456] LNet: Removed LNI 10.1.2.3@tcp [ 2152.101715] Key type .llcrypt unregistered [ 2152.107081] Key type ._llcrypt unregistered [ 2162.941956] Key type ._llcrypt registered [ 2162.952020] Key type .llcrypt registered [ 2164.484987] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:52:04 (1780332724) [ 2164.944922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2164.968491] alg: No test for adler32 (adler32-zlib) [ 2165.916387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.981253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2166.011916] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2166.757137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.57@tcp [ 2166.859503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg457-client@tcp [ 2173.296561] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:52:14 (1780332734) [ 2173.448222] LNet: 109054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2173.454879] LNetError: 109054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2173.476892] LNet: Removed LNI 192.168.204.57@tcp [ 2173.992504] Key type .llcrypt unregistered [ 2173.995841] Key type ._llcrypt unregistered [ 2179.045678] Key type ._llcrypt registered [ 2179.053769] Key type .llcrypt registered [ 2179.287331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2179.316779] alg: No test for adler32 (adler32-zlib) [ 2180.219564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2180.257090] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2180.286684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2186.075636] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:52:26 (1780332746) [ 2187.514639] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2190.472449] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:52:30 (1780332750) [ 2192.274805] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2194.188975] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:52:34 (1780332754) [ 2194.257508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.287299] LNet: 110468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.294491] LNetError: 110468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2194.319185] LNet: Removed LNI 192.168.204.57@tcp [ 2194.340384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.437118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2194.470988] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2195.431406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2195.553632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2203.357476] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:52:44 (1780332764) [ 2203.500067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.546508] LNet: 111184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.553434] LNetError: 111184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.568943] LNet: Removed LNI 192.168.204.57@tcp [ 2203.596840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.651259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2203.676632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2204.290458] Key type .llcrypt unregistered [ 2204.292573] Key type ._llcrypt unregistered [ 2214.604964] Key type ._llcrypt registered [ 2214.608813] Key type .llcrypt registered [ 2216.098329] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:52:56 (1780332776) [ 2216.513589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2216.524495] alg: No test for adler32 (adler32-zlib) [ 2217.468377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.545662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2217.613441] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2218.617636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2218.688823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2224.797430] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:53:05 (1780332785) [ 2225.090150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2225.136303] LNet: 112740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.153195] LNetError: 112740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2225.183438] LNet: Removed LNI 192.168.204.57@tcp [ 2225.225632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.287409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2225.332127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2231.717459] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:53:12 (1780332792) [ 2231.897232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.967334] LNet: 113314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.000385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.123600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2232.856341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2232.893096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2232.975652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2233.692157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2233.745428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2233.854246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2233.901547] LNet: 113353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.925307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.003128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2234.705729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2234.741991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2234.801882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2235.290882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2235.315934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2241.945558] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:53:22 (1780332802) [ 2242.069423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2242.136557] LNet: 113948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.195933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2242.310742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2242.357537] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2243.186347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2243.218526] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2244.258742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2244.344848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2244.675824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.702565] LNetError: 114234:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2244.712666] LNet: Removed LNI 192.168.204.57@tcp [ 2244.729295] LNet: Removed LNI 192.168.204.57@tcp2 [ 2244.765589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.885441] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2244.901624] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2244.910907] LNet: 114240:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2245.733611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2245.952807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.981529] LNetError: 114277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2245.988451] LNetError: 114277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2245.992383] LNet: Removed LNI 192.168.204.57@tcp [ 2246.005584] LNet: Removed LNI 192.168.204.57@tcp2 [ 2246.019513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.094123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2246.114787] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2246.791842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2246.816415] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2247.795760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2247.887914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2247.996961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2248.265415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.295625] LNet: 114562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.303515] LNet: 114562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2248.309337] LNetError: 114562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.318540] LNetError: 114562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2248.324971] LNet: Removed LNI 192.168.204.57@tcp [ 2249.386594] LNet: Removed LNI 192.168.204.57@tcp2 [ 2249.434779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2249.519874] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2249.526721] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2249.537476] LNet: 114568:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2250.264312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2250.510482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.533796] LNet: Removed LNI 192.168.204.57@tcp [ 2251.559654] LNet: Removed LNI 192.168.204.57@tcp2 [ 2251.586330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.695559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2251.757080] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2252.651385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2252.675409] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2253.673318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2253.756256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2254.038821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2254.067432] LNetError: 114884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2254.076709] LNetError: 114884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2254.098847] LNet: Removed LNI 192.168.204.57@tcp [ 2254.149870] LNet: Removed LNI 192.168.204.57@tcp2 [ 2254.215071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.287694] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2254.293356] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2254.298611] LNet: 114890:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2254.722764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2254.961803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.032252] LNet: Removed LNI 192.168.204.57@tcp [ 2256.057761] LNet: Removed LNI 192.168.204.57@tcp2 [ 2256.132708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.217559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.242512] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2257.170774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2257.186875] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2258.269364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2258.323064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2258.413262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2258.769810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.814306] LNet: 115212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.828392] LNet: 115212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2259.873186] LNet: Removed LNI 192.168.204.57@tcp [ 2260.909929] LNet: Removed LNI 192.168.204.57@tcp2 [ 2260.933617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.011306] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2261.019511] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2261.037975] LNet: 115218:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2261.492320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2261.873307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.913052] LNet: Removed LNI 192.168.204.57@tcp [ 2262.957809] LNet: Removed LNI 192.168.204.57@tcp2 [ 2262.992369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.088075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2263.111911] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2264.028063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2264.063279] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2265.091314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2265.118790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2265.194615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2265.603690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.643911] LNetError: 115552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.652318] LNetError: 115552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2265.660712] LNet: Removed LNI 192.168.204.57@tcp [ 2265.670668] LNet: Removed LNI 192.168.204.57@tcp2 [ 2265.707212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.777346] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2265.792508] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2265.800591] LNet: 115558:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2266.513799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2267.032762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.128740] LNet: Removed LNI 192.168.204.57@tcp [ 2268.147291] LNet: Removed LNI 192.168.204.57@tcp2 [ 2268.206937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.253557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2268.282415] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2269.232306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2269.252420] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2270.067935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2270.111216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2270.146086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2270.222830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2270.709266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.776548] LNet: Removed LNI 192.168.204.57@tcp [ 2272.807732] LNet: Removed LNI 192.168.204.57@tcp2 [ 2272.833631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.907685] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2272.912747] LNet: Added LNI 192.168.204.57@tcp2 [8/256/0/180] [ 2273.534586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2280.708480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2282.394804] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:54:03 (1780332843) [ 2282.508684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.540417] LNet: 116672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.550235] LNet: 116672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2282.565425] LNetError: 116672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.574487] LNetError: 116672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2282.583641] LNet: Removed LNI 192.168.204.57@tcp [ 2282.605724] LNet: Removed LNI 192.168.204.57@tcp2 [ 2282.699109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.933037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2282.954091] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2283.638985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2283.661676] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2284.488945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.503574] LNet: Removed LNI 192.168.204.57@tcp [ 2284.514771] LNet: Removed LNI 10.1.2.3@tcp [ 2284.533382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.598776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2284.624748] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2284.632664] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2285.061023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2285.094214] LNet: Removed LNI 192.168.204.57@tcp [ 2285.109073] LNet: Removed LNI 10.1.2.3@tcp [ 2291.239623] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:54:11 (1780332851) [ 2291.480565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2291.520307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2291.646043] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2291.723710] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2291.727909] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2291.732871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2294.106290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2294.125463] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2294.840169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2294.872373] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2296.721141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.57@tcp [ 2296.766286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2296.829321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.57@tcp -m PUT -e local_timeout -r 1 [ 2296.923821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2296.994456] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2296.999759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2297.280214] LNetError: 117618:0:(lib-move.c:4710:lnet_parse()) 192.168.204.57@tcp, src 192.168.204.57@tcp: Dropping PUT (error -108 looking up sender) [ 2299.040180] LNet: Removed LNI 192.168.204.57@tcp [ 2299.051863] LNet: Removed LNI 10.1.2.3@tcp [ 2299.539753] Key type .llcrypt unregistered [ 2299.542172] Key type ._llcrypt unregistered [ 2311.468784] Key type ._llcrypt registered [ 2311.470910] Key type .llcrypt registered [ 2312.471412] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:54:33 (1780332873) [ 2313.270314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2313.294475] alg: No test for adler32 (adler32-zlib) [ 2314.223455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.581675] LNet: 118931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.161985] Key type .llcrypt unregistered [ 2315.169468] Key type ._llcrypt unregistered [ 2320.975655] Key type ._llcrypt registered [ 2320.978041] Key type .llcrypt registered [ 2321.123762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2321.136767] alg: No test for adler32 (adler32-zlib) [ 2322.080461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2322.120572] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2329.196597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 2329.357960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2329.926861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.157@tcp [ 2330.142317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2330.191351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2330.256091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2330.325593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2330.336280] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.157@tcp [ 2330.384116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2330.436324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2330.486047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2330.543084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2330.562777] LNet: 119379:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2330.578204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2330.605397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2330.636296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2330.675221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2330.708783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2330.755375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2331.338306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2331.537454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2331.586437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2331.627674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2331.658801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2331.700946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2331.771218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2331.821977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2331.859350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2331.879501] LNet: 119496:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2331.896499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2331.920448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2331.948907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2331.980418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2332.005867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2332.038698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2332.676462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.157@tcp [ 2332.959499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2332.996466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2333.037969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2333.082148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2333.137135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2333.171586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2333.225936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2333.292909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2333.313815] LNet: 119614:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2333.354492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2333.384806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2333.444495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2333.477617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2333.513611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2333.575513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2334.265855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2334.660974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2334.700839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2334.770663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2334.807937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2334.840316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2334.880929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2334.911961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2334.942851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2334.984697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2335.022324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2335.078965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2335.121508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2335.154721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2335.186889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2335.540633] LNet: 119777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.551684] LNetError: 119777:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.572813] LNet: Removed LNI 192.168.204.57@tcp [ 2335.899905] Key type .llcrypt unregistered [ 2335.902210] Key type ._llcrypt unregistered [ 2352.013280] Key type ._llcrypt registered [ 2352.016429] Key type .llcrypt registered [ 2353.408695] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:55:13 (1780332913) [ 2353.780279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2353.790419] alg: No test for adler32 (adler32-zlib) [ 2354.707870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.181926] LNet: 120665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.692924] Key type .llcrypt unregistered [ 2355.695131] Key type ._llcrypt unregistered [ 2362.380247] Key type ._llcrypt registered [ 2362.386922] Key type .llcrypt registered [ 2362.702304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2362.738095] alg: No test for adler32 (adler32-zlib) [ 2363.689090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2363.730646] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2371.542451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.157@tcp [ 2382.583743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.636812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2382.673902] LNet: Added LNI 192.168.204.57@tcp1 [8/256/0/180] [ 2383.707332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2384.376808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.157@tcp [ 2384.678247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2384.712899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2384.741256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2384.767392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2384.809305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2384.844336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2384.907609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2384.975466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2385.009820] LNet: 121318:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2385.036331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2385.121057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2385.166511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2385.248307] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.157@tcp [ 2385.248522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2385.360565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2385.426337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2386.261608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2386.536161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2386.578714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2386.613046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2386.640941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2386.675976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2386.723320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2386.781886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2386.812707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2386.836861] LNet: 121435:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2386.873300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2386.912894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2386.947987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2386.993805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2387.045399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2387.095536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2387.770750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.157@tcp [ 2388.058746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2388.095470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2388.124356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2388.153614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2388.189628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2388.242730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2388.276984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2388.333803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2388.383584] LNet: 121552:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2388.406272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2388.443741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2388.470870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2388.527806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2388.589408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2388.630713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2389.370573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2389.619865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2389.653881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2389.683722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2389.709623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2389.739791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2389.774915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2389.804902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2389.838848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2389.864570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2389.900077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2389.935461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2389.980849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2390.037301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2390.080136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2394.626508] LNet: 121750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2394.642396] LNetError: 121750:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.649450] LNet: Removed LNI 192.168.204.57@tcp [ 2394.662071] LNet: Removed LNI 192.168.204.57@tcp1 [ 2395.082715] Key type .llcrypt unregistered [ 2395.084585] Key type ._llcrypt unregistered [ 2411.045731] Key type ._llcrypt registered [ 2411.048811] Key type .llcrypt registered [ 2412.401343] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:56:13 (1780332973) [ 2419.567891] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:56:20 (1780332980) [ 2426.100365] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:56:27 (1780332987) [ 2433.739805] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:56:34 (1780332994) [ 2440.280524] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:56:41 (1780333001) [ 2447.015874] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:56:48 (1780333008) [ 2454.418694] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:56:54 (1780333014) [ 2461.334745] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:57:02 (1780333022) [ 2468.527804] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:57:09 (1780333029) [ 2476.373643] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:57:16 (1780333036) [ 2483.268480] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:57:23 (1780333043) [ 2490.653251] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:57:31 (1780333051) [ 2496.784175] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:57:37 (1780333057) [ 2502.800263] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:57:43 (1780333063) [ 2509.620973] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:57:50 (1780333070) [ 2516.033451] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:57:57 (1780333077) [ 2523.245502] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:58:03 (1780333083) [ 2530.949609] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:58:11 (1780333091) [ 2531.461446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2531.479848] alg: No test for adler32 (adler32-zlib) [ 2532.477378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2532.680444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2532.710028] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2533.970853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2533.993456] LNet: 132461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.000129] LNetError: 132461:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2534.016959] LNet: Removed LNI 192.168.204.57@tcp [ 2534.035551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2534.096347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2534.120761] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2542.200435] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:58:22 (1780333102) [ 2549.976766] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:58:30 (1780333110) [ 2558.355413] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:58:38 (1780333118) [ 2565.782691] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:58:46 (1780333126) [ 2572.997973] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:58:53 (1780333133) [ 2580.157684] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:59:00 (1780333140) [ 2587.334759] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:59:08 (1780333148) [ 2594.571079] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:59:15 (1780333155) [ 2601.035083] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:59:21 (1780333161) [ 2607.710785] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:59:28 (1780333168) [ 2613.325079] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:59:34 (1780333174) [ 2619.634604] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:59:40 (1780333180) [ 2627.927919] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2280 sec ========== 12:59:47 (1780333187) [ 2628.392830] LNet: 139929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.403977] LNetError: 139929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.446167] LNet: Removed LNI 192.168.204.57@tcp [ 2629.188796] Key type .llcrypt unregistered [ 2629.192820] Key type ._llcrypt unregistered [ 2642.833266] Key type ._llcrypt registered [ 2642.836543] Key type .llcrypt registered [ 2643.211631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2643.224139] alg: No test for adler32 (adler32-zlib) [ 2644.461858] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 2644.761658] LNet: Added LNI 192.168.204.57@tcp [8/256/0/180] [ 2646.408474] Key type lgssc registered [ 2647.771570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2714.871887] Lustre: Mounted lustre-client [ 2721.472074] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2729.453978] LustreError: 142390:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff992fcb0b6800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2729.535840] Lustre: Unmounted lustre-client [ 2751.815527] Key type lgssc unregistered [ 2752.013854] LNet: 142871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2752.023172] LNetError: 142871:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2752.045290] LNet: Removed LNI 192.168.204.57@tcp [ 2752.531315] Key type .llcrypt unregistered [ 2752.535252] Key type ._llcrypt unregistered