[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 425431071 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003300] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.010000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011000] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.012064] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.014092] *** VALIDATE selinux *** [ 0.024000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028124] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031112] *** VALIDATE tmpfs *** [ 0.032739] *** VALIDATE proc *** [ 0.035121] *** VALIDATE cgroup *** [ 0.036012] *** VALIDATE cgroup2 *** [ 0.038532] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.042012] Speculative Store Bypass: Vulnerable [ 0.044966] debug: unmapping init [mem 0xffffffffbac59000-0xffffffffbac60fff] [ 0.047394] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048949] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050024] ... version: 2 [ 0.051015] ... bit width: 48 [ 0.052016] ... generic registers: 4 [ 0.053015] ... value mask: 0000ffffffffffff [ 0.054016] ... max period: 00007fffffffffff [ 0.055019] ... fixed-purpose events: 3 [ 0.056015] ... event mask: 000000070000000f [ 0.058374] rcu: Hierarchical SRCU implementation. [ 0.062557] smp: Bringing up secondary CPUs ... [ 0.063714] x86: Booting SMP configuration: [ 0.064046] .... node #0, CPUs: #1 #2 #3 [ 0.076022] smp: Brought up 1 node, 4 CPUs [ 0.078022] smpboot: Max logical packages: 1 [ 0.079025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142418] node 0 deferred pages initialised in 60ms [ 0.151011] devtmpfs: initialized [ 0.152276] x86/mm: Memory block size: 128MB [ 0.155033] gcov: version magic: 0x41383552 [ 0.161054] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163492] pinctrl core: initialized pinctrl subsystem [ 0.164522] [ 0.165023] ************************************************************* [ 0.166018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167019] ** ** [ 0.168018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169021] ** ** [ 0.170025] ** This means that this kernel is built to expose internal ** [ 0.171116] ** IOMMU data structures, which may compromise security on ** [ 0.172088] ** your system. ** [ 0.173016] ** ** [ 0.174020] ** If you see this message and you are not debugging the ** [ 0.175020] ** kernel, report this immediately to your vendor! ** [ 0.176021] ** ** [ 0.177023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178026] ************************************************************* [ 0.180418] NET: Registered protocol family 16 [ 0.182345] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183101] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187023] cpuidle: using governor menu [ 0.191398] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193393] PCI: Using configuration type 1 for base access [ 0.194171] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.207106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.221481] cryptd: max_cpu_qlen set to 1000 [ 0.224000] ACPI: Added _OSI(Module Device) [ 0.224000] ACPI: Added _OSI(Processor Device) [ 0.226023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242853] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250821] ACPI: Interpreter enabled [ 0.251093] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252022] ACPI: Using IOAPIC for interrupt routing [ 0.253317] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.254939] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.275054] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.283116] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.284025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285164] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.289286] acpiphp: Slot [2] registered [ 0.290297] acpiphp: Slot [5] registered [ 0.291761] acpiphp: Slot [6] registered [ 0.292263] acpiphp: Slot [3] registered [ 0.293202] acpiphp: Slot [4] registered [ 0.294131] acpiphp: Slot [7] registered [ 0.295150] acpiphp: Slot [8] registered [ 0.296273] acpiphp: Slot [9] registered [ 0.297154] acpiphp: Slot [10] registered [ 0.298368] acpiphp: Slot [11] registered [ 0.299135] acpiphp: Slot [12] registered [ 0.300433] acpiphp: Slot [13] registered [ 0.301135] acpiphp: Slot [14] registered [ 0.302148] acpiphp: Slot [15] registered [ 0.303115] acpiphp: Slot [16] registered [ 0.304224] acpiphp: Slot [17] registered [ 0.305157] acpiphp: Slot [18] registered [ 0.306146] acpiphp: Slot [19] registered [ 0.307373] acpiphp: Slot [20] registered [ 0.308200] acpiphp: Slot [21] registered [ 0.309194] acpiphp: Slot [22] registered [ 0.310197] acpiphp: Slot [23] registered [ 0.311160] acpiphp: Slot [24] registered [ 0.312139] acpiphp: Slot [25] registered [ 0.313279] acpiphp: Slot [26] registered [ 0.314137] acpiphp: Slot [27] registered [ 0.315142] acpiphp: Slot [28] registered [ 0.316123] acpiphp: Slot [29] registered [ 0.317955] acpiphp: Slot [30] registered [ 0.319394] acpiphp: Slot [31] registered [ 0.321086] PCI host bridge to bus 0000:00 [ 0.326033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.330024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.344024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.349163] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.352354] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.356846] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.362448] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.376015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.380000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.386031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.394017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.400031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.405852] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.412752] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.419054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.424109] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 11718 usecs [ 0.427099] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.441023] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.457022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.464019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.476803] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.485020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.492000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.507023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.519000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.526018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.532021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.547000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.563292] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.566580] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.568446] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.571481] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.575446] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.580058] iommu: Default domain type: Passthrough [ 0.581500] SCSI subsystem initialized [ 0.584961] ACPI: bus type USB registered [ 0.589165] usbcore: registered new interface driver usbfs [ 0.593108] usbcore: registered new interface driver hub [ 0.597121] usbcore: registered new device driver usb [ 0.602215] pps_core: LinuxPPS API ver. 1 registered [ 0.604016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.608067] PTP clock support registered [ 0.610069] EDAC MC: Ver: 3.0.0 [ 0.611674] PCI: Using ACPI for IRQ routing [ 0.612000] NetLabel: Initializing [ 0.614012] NetLabel: domain hash size = 128 [ 0.615014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.618095] NetLabel: unlabeled traffic allowed by default [ 0.621386] vgaarb: loaded [ 0.623588] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.626013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.633000] clocksource: Switched to clocksource kvm-clock [ 0.816186] VFS: Disk quotas dquot_6.6.0 [ 0.823093] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.831747] *** VALIDATE ramfs *** [ 0.834576] *** VALIDATE hugetlbfs *** [ 0.838623] pnp: PnP ACPI init [ 0.841984] pnp: PnP ACPI: found 6 devices [ 0.874707] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.880109] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.883085] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.886321] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.891391] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.895397] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.899716] NET: Registered protocol family 2 [ 0.904084] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.920075] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.927660] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.949871] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.957333] TCP: Hash tables configured (established 65536 bind 65536) [ 0.964677] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.971435] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.974579] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.980421] NET: Registered protocol family 1 [ 0.984715] RPC: Registered named UNIX socket transport module. [ 0.989024] RPC: Registered udp transport module. [ 0.992182] RPC: Registered tcp transport module. [ 0.995150] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.998563] NET: Registered protocol family 44 [ 1.000685] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.004189] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.006973] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.009813] PCI: CLS 0 bytes, default 64 [ 1.011822] Unpacking initramfs... [ 4.336197] debug: unmapping init [mem 0xffff9a26fcc64000-0xffff9a26fffcffff] [ 4.349588] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.355046] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.370261] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.116657] Initialise system trusted keyrings [ 6.123646] Key type blacklist registered [ 6.131428] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.151654] zbud: loaded [ 6.159204] *** VALIDATE nfs *** [ 6.167586] *** VALIDATE nfs4 *** [ 6.171612] pstore: using deflate compression [ 6.176476] Platform Keyring initialized [ 6.506953] NET: Registered protocol family 38 [ 6.510819] Key type asymmetric registered [ 6.520025] Asymmetric key parser 'x509' registered [ 6.524735] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.534540] io scheduler mq-deadline registered [ 6.539477] io scheduler kyber registered [ 6.543917] io scheduler bfq registered [ 6.548525] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.553723] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.565860] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.575386] ACPI: Power Button [PWRF] [ 6.590501] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.601944] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.624458] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.663518] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.711860] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.728762] Non-volatile memory driver v1.3 [ 6.736647] Linux agpgart interface v0.103 [ 6.873333] virtio_blk virtio1: [vda] 146936 512-byte logical blocks (75.2 MB/71.7 MiB) [ 6.883072] vda: detected capacity change from 0 to 75231232 [ 6.978647] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.991363] vdb: detected capacity change from 0 to 1073741824 [ 7.014053] libphy: Fixed MDIO Bus: probed [ 7.043629] usbcore: registered new interface driver usbserial_generic [ 7.050024] usbserial: USB Serial support registered for generic [ 7.053975] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.061139] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.064433] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.068263] mousedev: PS/2 mouse device common for all mice [ 7.078738] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.080397] rtc_cmos 00:05: RTC can wake from S4 [ 7.096248] rtc_cmos 00:05: registered as rtc0 [ 7.098951] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.105196] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.106688] intel_pstate: CPU model not supported [ 7.108547] hid: raw HID events driver (C) Jiri Kosina [ 7.124615] usbcore: registered new interface driver usbhid [ 7.128669] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.131129] usbhid: USB HID core driver [ 7.148278] drop_monitor: Initializing network drop monitor service [ 7.154689] Initializing XFRM netlink socket [ 7.158336] NET: Registered protocol family 10 [ 7.166752] Segment Routing with IPv6 [ 7.170038] NET: Registered protocol family 17 [ 7.174135] mpls_gso: MPLS GSO support [ 7.185319] RAS: Correctable Errors collector initialized. [ 7.190991] AVX version of gcm_enc/dec engaged. [ 7.193882] AES CTR mode by8 optimization enabled [ 7.301669] sched_clock: Marking stable (7301582375, 0)->(9091724652, -1790142277) [ 7.316619] registered taskstats version 1 [ 7.322110] Loading compiled-in X.509 certificates [ 7.325636] zswap: loaded using pool lzo/zbud [ 7.380483] Key type big_key registered [ 7.412902] Key type encrypted registered [ 7.416519] ima: No TPM chip found, activating TPM-bypass! [ 7.423558] ima: Allocated hash algorithm: sha1 [ 7.431274] ima: No architecture policies found [ 7.437722] evm: Initialising EVM extended attributes: [ 7.444552] evm: security.selinux [ 7.447162] evm: security.ima [ 7.449379] evm: security.capability [ 7.451607] evm: HMAC attrs: 0x1 [ 7.456074] rtc_cmos 00:05: setting system clock to 2026-08-27 15:05:22 UTC (1787843122) [ 7.464194] debug: unmapping init [mem 0xffffffffbbc03000-0xffffffffbbdfffff] [ 7.468334] debug: unmapping init [mem 0xffffffffba982000-0xffffffffbac58fff] [ 7.479156] Write protecting the kernel read-only data: 28672k [ 7.493111] debug: unmapping init [mem 0xffffffffb9003000-0xffffffffb91fffff] [ 7.502235] debug: unmapping init [mem 0xffffffffb9914000-0xffffffffb99fffff] [ 7.603255] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.622189] systemd[1]: Detected virtualization kvm. [ 7.629437] systemd[1]: Detected architecture x86-64. [ 7.632648] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.691985] systemd[1]: No hostname configured. [ 7.697298] systemd[1]: Set hostname to . [ 7.704169] random: systemd: uninitialized urandom read (16 bytes read) [ 7.711265] systemd[1]: Initializing machine ID from random generator. [ 8.015198] random: systemd: uninitialized urandom read (16 bytes read) [ 8.017325] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 8.025474] random: systemd: uninitialized urandom read (16 bytes read) [ 8.028885] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 8.034319] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.446603] device-mapper: uevent: version 1.0.3 [ 9.450404] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.876366] virtio_net virtio0 ens2: renamed from eth0 [ 10.996635] random: fast init done [ 11.040975] scsi host0: ata_piix [ 11.047738] scsi host1: ata_piix [ 11.054668] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 11.061878] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.963424] random: crng init done [ 15.970607] random: 7 urandom warning(s) missed due to ratelimiting [ 16.009676] dracut-initqueue[576]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.060303] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.715484] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.559891] SELinux: Disabled at runtime. [ 21.681213] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.710851] systemd[1]: Detected virtualization kvm. [ 21.712915] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.418471] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.433336] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.446734] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.459714] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.478787] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.515909] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.544495] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ 24.100057] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.214485] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.537655] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.572443] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.943245] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.050936] EDAC sbridge: Ver: 1.1.2 [ 30.751990] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit)[ 31.335859] NFS: Registering the id_resolver key type [ 31.340933] Key type id_resolver registered [ 31.343854] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg314-client login: [ 66.202129] hrtimer: interrupt took 5200451 ns [ 84.728724] libcfs: loading out-of-tree module taints kernel. [ 84.846071] Key type ._llcrypt registered [ 84.848129] Key type .llcrypt registered [ 85.592224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 85.619683] alg: No test for adler32 (adler32-zlib) [ 87.170606] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 88.139539] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 89.919587] Key type lgssc registered [ 92.360869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 213.946233] Lustre: Mounted lustre-client - version 2.17.57_89_g2dd35b7 [ 219.140882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.425691] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing check_logdir /tmp/testlogs/ [ 239.600318] Lustre: lustre-OST0000-osc-ffff9a2742be4800: disconnect after 23s idle [ 240.946377] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing yml_node [ 245.910543] Lustre: DEBUG MARKER: Client: 2.17.57.89 [ 248.631131] Lustre: DEBUG MARKER: MDS: 2.17.57.89 [ 251.517930] Lustre: DEBUG MARKER: OSS: 2.17.57.89 [ 253.440327] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 11:09:26 EDT 2026 [ 269.363451] Lustre: Unmounted lustre-client [ 289.827658] Key type lgssc unregistered [ 290.047441] LNet: 8419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.054943] LNetError: 8419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.081191] LNet: Removed LNI 192.168.203.14@tcp [ 290.925168] Key type .llcrypt unregistered [ 290.928249] Key type ._llcrypt unregistered [ 316.935599] Key type ._llcrypt registered [ 316.941871] Key type .llcrypt registered [ 317.428676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 317.457825] alg: No test for adler32 (adler32-zlib) [ 318.442779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 318.567736] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 318.603247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 323.013832] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 323.402751] LNet: 9306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.413802] LNetError: 9306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 323.438860] LNet: Removed LNI 192.168.203.14@tcp [ 324.349368] Key type .llcrypt unregistered [ 324.356414] Key type ._llcrypt unregistered [ 331.160163] Key type ._llcrypt registered [ 331.168314] Key type .llcrypt registered [ 332.830251] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:10:46 (1787843446) [ 333.188964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 333.214713] alg: No test for adler32 (adler32-zlib) [ 334.228544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.329894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.350716] LNet: 9721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.378225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.488109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 334.536718] LNet: 9728:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 342.760510] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:10:55 (1787843455) [ 342.901026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.956420] LNet: 10299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.987160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.130250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 352.118728] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:11:05 (1787843465) [ 352.322843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.378486] LNet: 10893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.421214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.498198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 360.834406] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:11:14 (1787843474) [ 361.035110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.064160] LNet: 11476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.111854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.238785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 368.537669] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:11:22 (1787843482) [ 368.663946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.718171] LNet: 12059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.763967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.877296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 369.297807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 376.382611] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:11:29 (1787843489) [ 376.500199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.552862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.656615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 377.092651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 377.529740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 383.658489] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:11:37 (1787843497) [ 383.836521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.862749] LNet: 13252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.868918] LNet: 13252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 383.883988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.957128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 391.584994] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:11:44 (1787843504) [ 391.695568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.756562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.845652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 391.909063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 392.549531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 392.584927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 393.123054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 393.164306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 393.636881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 393.705586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 394.165207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 394.191535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.573638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 394.621552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 395.193977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 395.242540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 395.733613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 395.775125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 396.183352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 396.224203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 396.814790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 396.866464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 397.340397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 397.401534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 397.969628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 398.006394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.474869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 398.542198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 404.892470] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:11:58 (1787843518) [ 405.159554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.192581] LNet: 14518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.201674] LNet: 14518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 405.240020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.320654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 405.363307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 411.593988] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:12:05 (1787843525) [ 411.809990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.845449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.920979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 411.961809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 418.357983] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:12:11 (1787843531) [ 418.488419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.574912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.695655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 418.739247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 425.358471] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 11:12:19 (1787843539) [ 425.411524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.451906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.533742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 425.560752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 431.920909] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 11:12:25 (1787843545) [ 432.052960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.097584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.205718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 432.247574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 438.877196] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 11:12:32 (1787843552) [ 439.018727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.061810] LNet: 17442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.074462] LNet: 17442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 439.115812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.258602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 439.317945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 447.550568] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 11:12:40 (1787843560) [ 447.754352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.853453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.033932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.325407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 455.586549] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 11:12:48 (1787843568) [ 455.748408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.817976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.066581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.478620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.198975] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 11:12:58 (1787843578) [ 465.581317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.627746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.837913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.320887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 473.893911] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 11:13:07 (1787843587) [ 474.202386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.251384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.752432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 474.955840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 481.872415] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 11:13:15 (1787843595) [ 482.201943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.263426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.350727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.419388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 488.627366] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 11:13:22 (1787843602) [ 488.880195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.924442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.015715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 489.105726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.238690] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 11:13:29 (1787843609) [ 496.450565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.538281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.646810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 504.030947] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 11:13:37 (1787843617) [ 504.231666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.276706] LNet: 22299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.282496] LNet: 22299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 504.320304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.406721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 511.174250] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 11:13:44 (1787843624) [ 511.270367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.321823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.451226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 511.809168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 518.763782] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 11:13:52 (1787843632) [ 519.142665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.204317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.315494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 519.353496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 526.238634] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 11:13:59 (1787843639) [ 526.371441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.453950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.617826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 526.676174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 534.486576] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 11:14:07 (1787843647) [ 534.574642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.649640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.748609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 534.802277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 541.792790] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 11:14:15 (1787843655) [ 541.868501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.941381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.060774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 542.147846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 548.948268] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 11:14:22 (1787843662) [ 549.132706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.175619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.244117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 549.327591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 549.362157] LNetError: 25834:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 549.401166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 549.461847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 555.864973] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 11:14:29 (1787843669) [ 555.953645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.982917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.094245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.5RBU6eFWy1 [ 556.187649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.5RBU6eFWy1 [ 556.256864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.5RBU6eFWy1 [ 563.213391] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 11:14:36 (1787843676) [ 563.402591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.454547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.552929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 563.664881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 563.764038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 572.757765] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 11:14:45 (1787843685) [ 572.886592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.967591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.091230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 573.144701] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 575.379217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 577.326772] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 579.336893] Key type lgssc registered [ 580.977358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 585.899718] LNetError: 27595:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.14@tcp99 has no available nets [ 588.880608] Key type lgssc unregistered [ 589.177743] LNetError: 28707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.200225] LNet: Removed LNI 192.168.203.14@tcp [ 590.049957] Key type .llcrypt unregistered [ 590.056404] Key type ._llcrypt unregistered [ 597.247897] Key type ._llcrypt registered [ 597.251449] Key type .llcrypt registered [ 598.849235] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 11:15:12 (1787843712) [ 599.467402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 599.516205] alg: No test for adler32 (adler32-zlib) [ 600.523839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.631642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 604.055674] LNet: Message forwarding will be enabled in 2s [ 607.135567] LNet: Message forwarding enabled [ 607.603982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 607.632669] LNet: Message forwarding will stop in 2s to 6s [ 612.113850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 612.663516] LNet: Message forwarding disabled [ 614.488029] LNet: Message forwarding will be enabled in 2s [ 617.828199] LNet: Message forwarding enabled [ 621.808052] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 11:15:35 (1787843735) [ 621.904989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.935922] LNet: 30110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.303884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.393933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 629.468930] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 11:15:43 (1787843743) [ 629.791359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.813788] LNet: 30697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.842504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.927478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 629.967720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 630.007260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 630.040789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 630.077495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 630.114304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 630.154174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 630.189523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 630.230229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 630.259911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 630.299693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 630.335785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 630.365502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 630.392836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 630.420916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 630.450720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 630.488652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 630.528868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 630.569475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 630.599843] LNetError: 30740:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 637.135930] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 11:15:50 (1787843750) [ 637.368521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.386737] LNet: 31311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.402573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.473682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 643.592797] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 11:15:57 (1787843757) [ 645.552495] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg314-server [ 647.563462] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 11:16:01 (1787843761) [ 647.739494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.769765] LNet: 32242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.798096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.874278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.898759] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 648.563483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[15-19]@tcp [ 648.579703] LNet: 32375:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 648.630965] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.15@tcp is being used as a gateway but routing feature is not turned on [ 648.651389] LNetError: 32246:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.15@tcp has gone from up to down [ 649.646664] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.16@tcp is being used as a gateway but routing feature is not turned on [ 649.663526] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 650.668833] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.15@tcp is being used as a gateway but routing feature is not turned on [ 650.690749] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 652.708187] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.16@tcp is being used as a gateway but routing feature is not turned on [ 652.718756] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 656.286384] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 11:16:09 (1787843769) [ 656.813197] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.17@tcp is being used as a gateway but routing feature is not turned on [ 656.831186] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 658.261497] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg314-server [ 660.493649] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 11:16:13 (1787843773) [ 660.577045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.597752] LNet: 33305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.605166] LNetError: 33305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.670583] LNet: Removed LNI 192.168.203.14@tcp [ 661.720954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.794599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.819903] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 662.621127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[15-20/2]@tcp [ 662.646144] LNet: 33429:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 662.656546] LNet: 33429:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 662.685148] LNetError: 33316:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.15@tcp has gone from up to down [ 662.687239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[15-20/2]@tcp [ 662.700604] LNetError: 33316:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 669.525591] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 11:16:23 (1787843783) [ 669.640831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.674452] LNet: 34015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.690611] LNetError: 34015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.715768] LNet: Removed LNI 192.168.203.14@tcp [ 669.756773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.931556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 670.002725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.048891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.123867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.182629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.219595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.282154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 670.352417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.390817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.490787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.598058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.635960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.688114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 670.771735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.805246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.856178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.962687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.999380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.043883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 671.126589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.190214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.329094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.461891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.535846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.633813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 671.737556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.826221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.931371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.001300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.038873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.085793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 672.132506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.170367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.242730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.014523] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 11:16:32 (1787843792) [ 680.941931] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg314-server [ 682.852515] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 11:16:36 (1787843796) [ 684.895746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg314-server [ 686.781980] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 11:16:40 (1787843800) [ 687.006915] LNet: 35425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.012606] LNet: 35425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 687.776393] Key type .llcrypt unregistered [ 687.779336] Key type ._llcrypt unregistered [ 694.232658] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 694.384337] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 694.389899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 694.400398] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 696.952709] Key type ._llcrypt registered [ 696.954777] Key type .llcrypt registered [ 697.161136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 697.189768] alg: No test for adler32 (adler32-zlib) [ 698.158263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.248486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 698.272804] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 698.985710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 699.008049] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.960909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 700.985115] LNet: Removed LNI 10.1.2.3@tcp [ 701.141680] LNet: 36146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.153128] LNetError: 36146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.181083] LNet: Removed LNI 192.168.203.14@tcp [ 701.807614] Key type .llcrypt unregistered [ 701.810436] Key type ._llcrypt unregistered [ 713.221860] Key type ._llcrypt registered [ 713.226747] Key type .llcrypt registered [ 714.286558] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 11:17:08 (1787843828) [ 714.898331] Key type .llcrypt unregistered [ 714.902341] Key type ._llcrypt unregistered [ 719.921095] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 720.143048] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 720.152966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 720.159621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 722.796146] Key type ._llcrypt registered [ 722.811066] Key type .llcrypt registered [ 723.018979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 723.036977] alg: No test for adler32 (adler32-zlib) [ 723.996909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.041947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 724.075632] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 724.764639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 724.783294] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 727.267931] LNet: 37644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.276301] LNetError: 37644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 727.281883] LNet: Removed LNI 10.1.2.3@tcp [ 727.298427] LNet: Removed LNI 192.168.203.14@tcp [ 727.895417] Key type .llcrypt unregistered [ 727.901622] Key type ._llcrypt unregistered [ 738.516194] Key type ._llcrypt registered [ 738.522699] Key type .llcrypt registered [ 740.135842] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 11:17:33 (1787843853) [ 741.052455] Key type .llcrypt unregistered [ 741.058330] Key type ._llcrypt unregistered [ 749.298928] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 749.560948] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 749.570244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 749.579338] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 752.436885] Key type ._llcrypt registered [ 752.438771] Key type .llcrypt registered [ 752.649732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 752.668505] alg: No test for adler32 (adler32-zlib) [ 753.686203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.744392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 753.794716] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 756.850320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.873425] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.617895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 759.647792] LNet: Removed LNI 10.1.2.3@tcp [ 759.664456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 759.694662] LNet: Removed LNI 10.1.2.31@tcp [ 759.715084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 759.775688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.593765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 760.611101] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 761.702218] LNet: 39567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.706928] LNetError: 39567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 761.712238] LNet: Removed LNI 10.1.2.31@tcp [ 761.722327] LNet: Removed LNI 10.1.2.3@tcp [ 762.587103] Key type .llcrypt unregistered [ 762.597267] Key type ._llcrypt unregistered [ 772.803298] Key type ._llcrypt registered [ 772.806366] Key type .llcrypt registered [ 774.059593] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 11:18:07 (1787843887) [ 774.663767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 774.684645] alg: No test for adler32 (adler32-zlib) [ 775.635624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.719920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 775.744195] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 776.346931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.388798] LNet: 40525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.403434] LNetError: 40525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.424860] LNet: Removed LNI 192.168.203.14@tcp [ 776.451208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.536232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.14@tcp [ 776.576666] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 777.448181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.14@tcp [ 777.489296] LNet: Removed LNI 192.168.203.14@tcp [ 777.589353] LNet: 40660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.317438] Key type .llcrypt unregistered [ 778.322418] Key type ._llcrypt unregistered [ 790.506222] Key type ._llcrypt registered [ 790.509622] Key type .llcrypt registered [ 791.843572] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 11:18:25 (1787843905) [ 793.628321] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg314-server [ 795.556316] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 11:18:29 (1787843909) [ 796.133443] Key type .llcrypt unregistered [ 796.136837] Key type ._llcrypt unregistered [ 802.195090] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 802.288597] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 802.297502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 802.304048] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 803.791176] Key type ._llcrypt registered [ 803.793991] Key type .llcrypt registered [ 803.966696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.985373] alg: No test for adler32 (adler32-zlib) [ 804.909193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.966430] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 805.682399] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 807.975955] LNet: 42495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.983552] LNetError: 42495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.989942] LNet: Removed LNI 192.168.203.14@tcp1 [ 808.007932] LNet: Removed LNI 10.1.2.3@tcp2 [ 808.683641] Key type .llcrypt unregistered [ 808.689977] Key type ._llcrypt unregistered [ 820.705643] Key type ._llcrypt registered [ 820.707911] Key type .llcrypt registered [ 821.973967] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 11:18:55 (1787843935) [ 822.464231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.481542] alg: No test for adler32 (adler32-zlib) [ 823.389086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.455646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.492276] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 824.253886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.280621] LNet: 43468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.287214] LNetError: 43468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.310535] LNet: Removed LNI 192.168.203.14@tcp [ 824.331533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.413278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 824.454158] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 830.005368] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 11:19:03 (1787843943) [ 830.179941] LNet: 44057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.190052] LNetError: 44057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.219419] LNet: Removed LNI 192.168.203.14@tcp [ 830.931421] Key type .llcrypt unregistered [ 830.933828] Key type ._llcrypt unregistered [ 836.807133] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 836.933266] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 836.941802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 836.946660] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 839.576192] Key type ._llcrypt registered [ 839.579101] Key type .llcrypt registered [ 839.756397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 839.768529] alg: No test for adler32 (adler32-zlib) [ 840.768272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.865882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 840.906060] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 841.860253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 841.903500] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 843.898444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.912246] LNet: 44799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.916404] LNetError: 44799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 843.920345] LNet: Removed LNI 10.1.2.3@tcp [ 843.928190] LNet: Removed LNI 192.168.203.14@tcp [ 843.959933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.043510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 844.082892] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 844.088763] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 844.645280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.665954] LNet: 44826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.670809] LNetError: 44826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 844.676791] LNetError: 44826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 844.683567] LNet: Removed LNI 10.1.2.3@tcp [ 844.694046] LNet: Removed LNI 192.168.203.14@tcp [ 844.735788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.817819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 844.868911] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 844.880510] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 845.504476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 845.538751] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 846.506484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.556981] LNet: 44988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.566088] LNetError: 44988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 846.574049] LNetError: 44988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 846.586296] LNet: Removed LNI 10.1.2.3@tcp [ 846.593686] LNet: Removed LNI 192.168.203.14@tcp [ 846.609375] LNet: Removed LNI 192.168.203.14@tcp2 [ 846.684335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.803993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 846.851764] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 846.855412] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 846.863316] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 853.732597] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 11:19:27 (1787843967) [ 853.830664] LNet: 45576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.837043] LNetError: 45576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.842793] LNetError: 45576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 853.848912] LNet: Removed LNI 10.1.2.3@tcp [ 853.852855] LNet: Removed LNI 192.168.203.14@tcp [ 853.863222] LNet: Removed LNI 192.168.203.14@tcp2 [ 854.556771] Key type .llcrypt unregistered [ 854.559260] Key type ._llcrypt unregistered [ 860.499936] Key type ._llcrypt registered [ 860.503123] Key type .llcrypt registered [ 860.877781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 860.898717] alg: No test for adler32 (adler32-zlib) [ 861.882810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.959872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.999922] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 862.881420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.905771] LNet: 45983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.910926] LNetError: 45983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.927707] LNet: Removed LNI 192.168.203.14@tcp [ 862.963160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.031888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 863.051519] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 863.571184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.597328] LNet: 46014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.604414] LNetError: 46014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.628595] LNet: Removed LNI 192.168.203.14@tcp [ 863.649609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.709489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 863.743498] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 864.216257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.247578] LNet: Removed LNI 192.168.203.14@tcp [ 864.274097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.351953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 864.383065] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 864.927811] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 865.117898] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 865.121885] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 865.131154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 867.445624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.469390] LNet: 46116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.474595] LNet: 46116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 867.490471] LNetError: 46116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.501375] LNetError: 46116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 867.523391] LNet: Removed LNI 192.168.203.14@tcp [ 867.549468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.609655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 867.632276] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 873.617283] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 11:19:47 (1787843987) [ 873.732561] LNet: 46705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.741329] LNetError: 46705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.758908] LNet: Removed LNI 192.168.203.14@tcp [ 874.403718] Key type .llcrypt unregistered [ 874.409849] Key type ._llcrypt unregistered [ 879.351916] Key type ._llcrypt registered [ 879.355287] Key type .llcrypt registered [ 879.508304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.517818] alg: No test for adler32 (adler32-zlib) [ 880.438804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.503344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.528954] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 881.192661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.208939] LNet: 47116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.217484] LNetError: 47116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.236708] LNet: Removed LNI 192.168.203.14@tcp [ 881.255222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.304843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 881.334100] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 886.717428] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 11:20:00 (1787844000) [ 886.823912] LNet: 47702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.831994] LNetError: 47702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.851638] LNet: Removed LNI 192.168.203.14@tcp [ 887.499980] Key type .llcrypt unregistered [ 887.503667] Key type ._llcrypt unregistered [ 892.785722] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 892.998474] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 893.007944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 893.010605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 895.516188] Key type ._llcrypt registered [ 895.519124] Key type .llcrypt registered [ 895.670106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 895.683991] alg: No test for adler32 (adler32-zlib) [ 896.602332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.657352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.689233] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 897.414539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.433675] LNet: 48161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.439610] LNetError: 48161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.451353] LNet: Removed LNI 192.168.203.14@tcp [ 897.472074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.519500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 897.537370] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 902.810833] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 11:20:16 (1787844016) [ 902.949907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.976823] LNet: 48761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.981852] LNetError: 48761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.994210] LNet: Removed LNI 192.168.203.14@tcp [ 903.009464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.111915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 908.734557] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 11:20:22 (1787844022) [ 908.872787] LNet: 49337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.447722] Key type .llcrypt unregistered [ 909.450096] Key type ._llcrypt unregistered [ 915.119382] Key type ._llcrypt registered [ 915.123872] Key type .llcrypt registered [ 915.295771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 915.315192] alg: No test for adler32 (adler32-zlib) [ 916.234669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.283878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 916.310923] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 917.086534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.106492] LNet: 49745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.113491] LNetError: 49745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.133845] LNet: Removed LNI 192.168.203.14@tcp [ 917.151317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.215269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 917.239181] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 922.835630] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 11:20:36 (1787844036) [ 923.090090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.124768] LNet: 50338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.130801] LNetError: 50338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.147587] LNet: Removed LNI 192.168.203.14@tcp [ 923.168454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.247311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 928.092300] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 11:20:42 (1787844042) [ 929.228402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 930.886722] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 11:20:44 (1787844044) [ 932.334812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 933.772511] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 11:20:47 (1787844047) [ 934.940983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 936.607155] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 11:20:50 (1787844050) [ 938.038574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 939.728268] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 11:20:53 (1787844053) [ 941.167571] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 942.646991] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 11:20:56 (1787844056) [ 942.718757] LNet: 52381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.381782] Key type .llcrypt unregistered [ 943.384167] Key type ._llcrypt unregistered [ 948.865099] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.040706] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 949.058487] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 949.083814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.628752] Key type ._llcrypt registered [ 951.632243] Key type .llcrypt registered [ 951.828587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 951.856218] alg: No test for adler32 (adler32-zlib) [ 952.759990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.808871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 952.822698] LNet: Added LNI 192.168.203.14@tcp [8/512/0/180] [ 952.827182] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 952.829305] LNet: 52710:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 952.928943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.947556] LNet: 52729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.955541] LNetError: 52729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 952.961568] LNet: Removed LNI 10.1.2.3@tcp [ 952.966857] LNetError: 52729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.992964] LNet: Removed LNI 192.168.203.14@tcp [ 953.020775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.090472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.106853] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 953.196162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.218723] LNet: 52753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.224330] LNetError: 52753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.239814] LNet: Removed LNI 192.168.203.14@tcp [ 953.259194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.329698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.354724] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 953.365991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.371887] LNet: 52762:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 953.459990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.504956] LNet: 52780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.509671] LNetError: 52780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 953.515625] LNet: Removed LNI 10.1.2.3@tcp [ 953.518362] LNetError: 52780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.531269] LNet: Removed LNI 192.168.203.14@tcp [ 953.599111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.700668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.721306] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 953.725859] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.727909] LNet: 52788:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 953.862731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.880249] LNet: 52814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.890655] LNetError: 52814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 953.897952] LNet: Removed LNI 10.1.2.3@tcp [ 953.901106] LNetError: 52814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.912264] LNet: Removed LNI 192.168.203.14@tcp [ 953.928686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.987432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 954.029955] LNet: Added LNI 192.168.203.14@tcp [8/256/0/182] [ 954.042955] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 954.134259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.147981] LNet: 52842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.158599] LNetError: 52842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 954.163494] LNet: Removed LNI 10.1.2.3@tcp [ 954.165612] LNetError: 52842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.177429] LNet: Removed LNI 192.168.203.14@tcp [ 954.205098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.287704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 954.332358] LNet: Added LNI 192.168.203.14@tcp [8/256/0/182] [ 954.338845] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 954.341067] LNet: 52850:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 954.422856] LNet: Removed LNI 10.1.2.3@tcp [ 954.437384] LNet: Removed LNI 192.168.203.14@tcp [ 954.950917] Key type .llcrypt unregistered [ 954.954626] Key type ._llcrypt unregistered [ 964.377231] Key type ._llcrypt registered [ 964.379311] Key type .llcrypt registered [ 965.658755] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 11:21:19 (1787844079) [ 967.148991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 968.842809] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 11:21:22 (1787844082) [ 969.196985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.224966] alg: No test for adler32 (adler32-zlib) [ 970.165371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.233830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 970.250402] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 970.945885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.14 [ 970.963422] LNet: 54118:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 970.973424] LNet: Removed LNI 192.168.203.14@tcp [ 976.883562] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 11:21:30 (1787844090) [ 976.972179] LNet: 54682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.983155] LNetError: 54682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.998285] LNet: Removed LNI 192.168.203.14@tcp [ 977.561831] Key type .llcrypt unregistered [ 977.564664] Key type ._llcrypt unregistered [ 982.726619] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 982.923868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 985.556578] Key type ._llcrypt registered [ 985.558825] Key type .llcrypt registered [ 985.676196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 985.685396] alg: No test for adler32 (adler32-zlib) [ 986.990549] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 992.603455] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 11:21:46 (1787844106) [ 992.780669] LNet: 55605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.792117] LNetError: 55605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 992.810444] LNet: Removed LNI 10.1.2.3@tcp [ 993.466931] Key type .llcrypt unregistered [ 993.468784] Key type ._llcrypt unregistered [ 999.085808] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 999.360588] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 999.363737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.367453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1002.185675] Key type ._llcrypt registered [ 1002.193723] Key type .llcrypt registered [ 1002.344703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.387725] alg: No test for adler32 (adler32-zlib) [ 1003.335101] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.790987] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 11:22:03 (1787844123) [ 1011.025877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1012.837462] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 11:22:06 (1787844126) [ 1013.078900] LNet: 56794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.089102] LNetError: 56794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1013.126533] LNet: Removed LNI 10.1.2.3@tcp [ 1013.793443] Key type .llcrypt unregistered [ 1013.797596] Key type ._llcrypt unregistered [ 1020.135113] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1020.418353] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1020.429467] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1020.435164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1023.262984] Key type ._llcrypt registered [ 1023.268128] Key type .llcrypt registered [ 1023.488886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1023.505397] alg: No test for adler32 (adler32-zlib) [ 1024.445971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.571182] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1032.659257] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 11:22:26 (1787844146) [ 1033.107169] LNet: 58027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.121151] LNetError: 58027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1033.131795] LNet: Removed LNI 10.1.2.3@tcp [ 1033.888549] Key type .llcrypt unregistered [ 1033.892575] Key type ._llcrypt unregistered [ 1039.608927] Key type ._llcrypt registered [ 1039.611936] Key type .llcrypt registered [ 1039.843472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1039.861942] alg: No test for adler32 (adler32-zlib) [ 1040.865567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1040.893926] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1049.644892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 1050.575113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_interrupt -r 1 [ 1050.648725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_interrupt -r 1 [ 1050.691225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1050.715052] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1051.047449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.094353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1051.158600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.170278] LNet: 1 local NIs in recovery (showing 1): 192.168.203.14@tcp [ 1051.239767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.270258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.310305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.355411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.406661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.426306] LNet: 58468:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1051.446297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.474623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1051.502617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1051.537688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1051.577540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1051.611402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.645124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1052.178771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_dropped -r 1 [ 1052.216236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_dropped -r 1 [ 1052.263750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1052.289772] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1052.295969] LNet: Skipped 1 previous similar message [ 1052.556118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.590717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.617247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.675947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.702575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.742260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.784424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.850397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.870300] LNet: 58577:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1052.911735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.959362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.004183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.057830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.121646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.153243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.174862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1053.834417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_aborted -r 1 [ 1053.863328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_aborted -r 1 [ 1053.896899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1053.926911] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1054.229611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.253601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1054.283649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.304986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.334179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.365378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.395984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.434312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.454810] LNet: 58686:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1054.475855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.501849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.529152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.550738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.579747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.609707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.636691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1055.173917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_no_route -r 1 [ 1055.219585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_no_route -r 1 [ 1055.253608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1055.497183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.533735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1055.567078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.599869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.635543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.707481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.761824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.830565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.884437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1055.939367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.016553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.075355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.179643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.230710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.285253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1057.096257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_timeout -r 1 [ 1057.159971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_timeout -r 1 [ 1057.197633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1057.213086] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1057.215926] LNet: Skipped 2 previous similar messages [ 1057.525083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.583986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1057.655265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.732343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.788875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.850297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.918743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.952974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.964872] LNet: 58905:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1057.970965] LNet: 58905:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1057.983958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1058.027041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.070114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.125929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.168481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.206186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.296195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1059.005933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_error -r 1 [ 1059.040919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_error -r 1 [ 1059.082747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1059.381766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.425853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1059.453741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1059.480917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1059.507878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1059.543339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1059.579788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1059.627874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1059.651765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1059.692354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1059.752561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1059.814967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1059.889522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1059.944960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.003771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1060.399594] LNet: 59059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.416679] LNetError: 59059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.441780] LNet: Removed LNI 192.168.203.14@tcp [ 1060.979603] Key type .llcrypt unregistered [ 1060.984263] Key type ._llcrypt unregistered [ 1077.605467] Key type ._llcrypt registered [ 1077.610028] Key type .llcrypt registered [ 1079.204849] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 11:23:12 (1787844192) [ 1080.270350] Key type .llcrypt unregistered [ 1080.272973] Key type ._llcrypt unregistered [ 1087.697756] Key type ._llcrypt registered [ 1087.702692] Key type .llcrypt registered [ 1087.999858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1088.021319] alg: No test for adler32 (adler32-zlib) [ 1088.983260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1089.020578] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1097.505147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 1110.304867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1110.353858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1110.408573] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1112.071665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_interrupt -r 1 [ 1112.130280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_interrupt -r 1 [ 1112.182436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_interrupt -r 1 [ 1112.233182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_interrupt -r 1 [ 1112.281146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_interrupt -r 1 [ 1112.333549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_interrupt -r 1 [ 1112.377603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_interrupt -r 1 [ 1112.424225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_interrupt -r 1 [ 1112.478950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1112.514045] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1112.517750] LNet: 1 local NIs in recovery (showing 1): 192.168.203.14@tcp [ 1112.811932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.855245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1112.909211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.955920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.986355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.008634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.033863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.053841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.069909] LNet: 60533:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.082243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.101730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.126869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.152533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.186904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.214790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.253773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1113.731582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_dropped -r 1 [ 1113.775561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_dropped -r 1 [ 1113.814706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_dropped -r 1 [ 1113.847660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_dropped -r 1 [ 1113.885239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_dropped -r 1 [ 1113.941206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_dropped -r 1 [ 1113.988219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_dropped -r 1 [ 1114.025933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_dropped -r 1 [ 1114.073198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1114.098626] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1114.104996] LNet: Skipped 2 previous similar messages [ 1114.310624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.346506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1114.378432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.409385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.447670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.472447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.499621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.522679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.538971] LNet: 60654:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.555669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.578729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.603854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.626257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.654727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.684752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.726908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.227064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_aborted -r 1 [ 1115.257763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_aborted -r 1 [ 1115.291570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_aborted -r 1 [ 1115.337877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_aborted -r 1 [ 1115.390860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_aborted -r 1 [ 1115.445609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_aborted -r 1 [ 1115.478941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_aborted -r 1 [ 1115.529991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_aborted -r 1 [ 1115.579597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1115.596817] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1115.602793] LNet: Skipped 2 previous similar messages [ 1115.848919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.902917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1115.940881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.973923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.023824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.058290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.091833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.136699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.175477] LNet: 60776:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.204102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.260278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.293388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.332381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.380802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.436718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.476964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1116.984456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_no_route -r 1 [ 1117.031780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_no_route -r 1 [ 1117.064281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_no_route -r 1 [ 1117.097599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_no_route -r 1 [ 1117.141085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_no_route -r 1 [ 1117.176747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_no_route -r 1 [ 1117.234955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_no_route -r 1 [ 1117.293285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_no_route -r 1 [ 1117.333740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1117.583769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.611206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1117.654182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.692396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.720422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.755624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.859031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.904445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.951867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.004177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.029467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.056077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.096285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.148405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.195857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1118.756297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_timeout -r 1 [ 1118.811139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_timeout -r 1 [ 1118.848436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_timeout -r 1 [ 1118.887759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_timeout -r 1 [ 1118.936520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_timeout -r 1 [ 1118.982262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_timeout -r 1 [ 1119.056105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_timeout -r 1 [ 1119.133657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_timeout -r 1 [ 1119.193990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1119.216099] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1119.222304] LNet: Skipped 5 previous similar messages [ 1119.460635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.492259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.531424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.564725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.594336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.641948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.683793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.721689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.754468] LNet: 61018:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1119.763816] LNet: 61018:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1119.788565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.832091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.888861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.927453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.979731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.033422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.100826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.679219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_error -r 1 [ 1120.750392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_error -r 1 [ 1120.800838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_error -r 1 [ 1120.843045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_error -r 1 [ 1120.895391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_error -r 1 [ 1120.942679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_error -r 1 [ 1121.011927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_error -r 1 [ 1121.056729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_error -r 1 [ 1121.090823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1121.325783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.357833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.380623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.406115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.444230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.480110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.526901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.565390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.606752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.666948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.705718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.745224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.780233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.823113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.897084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1127.114986] LNet: 61220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.119247] LNetError: 61220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.125515] LNet: Removed LNI 192.168.203.14@tcp [ 1127.136788] LNet: Removed LNI 192.168.203.14@tcp1 [ 1127.559390] Key type .llcrypt unregistered [ 1127.569133] Key type ._llcrypt unregistered [ 1144.113148] Key type ._llcrypt registered [ 1144.116653] Key type .llcrypt registered [ 1145.661520] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 11:24:19 (1787844259) [ 1146.368583] Key type .llcrypt unregistered [ 1146.372425] Key type ._llcrypt unregistered [ 1152.322371] Key type ._llcrypt registered [ 1152.324393] Key type .llcrypt registered [ 1152.531695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1152.548405] alg: No test for adler32 (adler32-zlib) [ 1153.484764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1153.508114] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1160.807922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 1161.599729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e remote_dropped -r 1 [ 1161.654100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e remote_dropped -r 1 [ 1161.735067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1161.753612] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1161.957038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.015364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1162.053322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.088333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.113645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.150585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.187309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.228625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.252031] LNet: 62474:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.277603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.314486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.345811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.385240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.418437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.454369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.495974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.968476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e remote_error -r 1 [ 1163.024211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e remote_error -r 1 [ 1163.072804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1163.106078] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1163.323503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.374664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.432981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.476630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.530768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.573154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.612413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.655167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.683332] LNet: 62583:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.711043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.744071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.791334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.831807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.870362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.920942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.949212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1164.604674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e remote_timeout -r 1 [ 1164.646390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e remote_timeout -r 1 [ 1164.676278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1164.691789] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1164.948307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.983937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.013690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.056346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.114509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.152517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.183745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.225804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.262394] LNet: 62692:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.306747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.356486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.396518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.433611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.510201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.562125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.624343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1165.878236] LNet: 62737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.882976] LNetError: 62737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.904522] LNet: Removed LNI 192.168.203.14@tcp [ 1166.310145] Key type .llcrypt unregistered [ 1166.313042] Key type ._llcrypt unregistered [ 1182.312263] Key type ._llcrypt registered [ 1182.314691] Key type .llcrypt registered [ 1183.778321] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 11:24:57 (1787844297) [ 1184.699611] Key type .llcrypt unregistered [ 1184.701992] Key type ._llcrypt unregistered [ 1190.881945] Key type ._llcrypt registered [ 1190.884899] Key type .llcrypt registered [ 1191.126757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.156765] alg: No test for adler32 (adler32-zlib) [ 1192.086562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1192.140833] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1199.097500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 1207.327966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.375887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1207.418669] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1209.288990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e remote_dropped -r 1 [ 1209.330656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e remote_dropped -r 1 [ 1209.371570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e remote_dropped -r 1 [ 1209.420861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e remote_dropped -r 1 [ 1209.473168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e remote_dropped -r 1 [ 1209.547681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e remote_dropped -r 1 [ 1209.602117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e remote_dropped -r 1 [ 1209.652051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e remote_dropped -r 1 [ 1209.682664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1209.696505] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1209.701894] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.114@tcp [ 1209.902400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.964369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1210.021312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1210.062345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1210.094801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1210.122085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1210.156715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1210.187718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1210.205945] LNet: 64208:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1210.223236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1210.249791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1210.283193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1210.313308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1210.383293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.445882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1210.496304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1210.542681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.957844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e remote_error -r 1 [ 1211.018153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e remote_error -r 1 [ 1211.056754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e remote_error -r 1 [ 1211.094916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e remote_error -r 1 [ 1211.121693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e remote_error -r 1 [ 1211.150823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e remote_error -r 1 [ 1211.178394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e remote_error -r 1 [ 1211.216631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e remote_error -r 1 [ 1211.261041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1211.300472] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1211.306652] LNet: Skipped 2 previous similar messages [ 1211.553680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.628236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1211.672848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.740176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.792679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.816168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.843214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.872709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.888850] LNet: 64332:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.921404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.954751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.015694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.057245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.107753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.150635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.205687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1212.257224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.838561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e remote_timeout -r 1 [ 1212.882881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e remote_timeout -r 1 [ 1212.945565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e remote_timeout -r 1 [ 1212.991414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e remote_timeout -r 1 [ 1213.042707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e remote_timeout -r 1 [ 1213.094152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e remote_timeout -r 1 [ 1213.130825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e remote_timeout -r 1 [ 1213.165482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e remote_timeout -r 1 [ 1213.198675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.114@tcp [ 1213.218555] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1213.431558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1213.462794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1213.495414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.520233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.554556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.581891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.612065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.671519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.701368] LNet: 64455:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.751784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.802457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.829198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.855945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.884879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.939896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.985088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.028709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.691396] LNet: 64537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.699553] LNetError: 64537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.714747] LNet: Removed LNI 192.168.203.14@tcp [ 1219.726231] LNet: Removed LNI 192.168.203.14@tcp1 [ 1220.194709] Key type .llcrypt unregistered [ 1220.196795] Key type ._llcrypt unregistered [ 1234.409679] Key type ._llcrypt registered [ 1234.412453] Key type .llcrypt registered [ 1235.699126] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:25:49 (1787844349) [ 1236.944642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1238.638955] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:25:52 (1787844352) [ 1239.513399] Key type .llcrypt unregistered [ 1239.515487] Key type ._llcrypt unregistered [ 1244.368392] Key type ._llcrypt registered [ 1244.370663] Key type .llcrypt registered [ 1244.548951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1244.580283] alg: No test for adler32 (adler32-zlib) [ 1245.456962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1245.484766] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1251.795969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 1252.474568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e network_timeout -r 1 [ 1252.510557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e network_timeout -r 1 [ 1252.546731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 1252.571379] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1252.640390] LNet: 1 local NIs in recovery (showing 1): 192.168.203.14@tcp [ 1252.781129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.814911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1252.857619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1252.900616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1252.938147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.975138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1253.018236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1253.074120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1253.095614] LNet: 66083:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1253.132273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1253.181953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1253.212497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.259633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.297719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.340493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1253.617359] LNet: 66126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.625091] LNetError: 66126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.648441] LNet: Removed LNI 192.168.203.14@tcp [ 1253.994481] Key type .llcrypt unregistered [ 1253.997449] Key type ._llcrypt unregistered [ 1270.492703] Key type ._llcrypt registered [ 1270.494828] Key type .llcrypt registered [ 1270.661155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1270.671664] alg: No test for adler32 (adler32-zlib) [ 1271.531684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1271.560323] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1279.393757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 1289.202961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1289.236556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1289.259818] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1290.784844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e network_timeout -r 1 [ 1290.831868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e network_timeout -r 1 [ 1290.912869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e network_timeout -r 1 [ 1291.022799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e network_timeout -r 1 [ 1291.130782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e network_timeout -r 1 [ 1291.186269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e network_timeout -r 1 [ 1291.266975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e network_timeout -r 1 [ 1291.330511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e network_timeout -r 1 [ 1291.395283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 1291.452615] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1291.773736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1291.817089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1291.853486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1291.903982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1291.972024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.024365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.082896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.128295] LNet: 1 local NIs in recovery (showing 1): 192.168.203.14@tcp [ 1292.131130] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1292.135164] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.114@tcp [ 1292.145444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1292.198105] LNet: 67009:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1292.229697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1292.267775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1292.307152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1292.357488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1292.407145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1292.446896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1298.270102] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1298.283467] LNet: Skipped 1 previous similar message [ 1298.728078] LNet: 67087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.738384] LNetError: 67087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.747893] LNet: Removed LNI 192.168.203.14@tcp [ 1298.761045] LNet: Removed LNI 192.168.203.14@tcp1 [ 1299.255580] Key type .llcrypt unregistered [ 1299.259970] Key type ._llcrypt unregistered [ 1316.695332] Key type ._llcrypt registered [ 1316.697114] Key type .llcrypt registered [ 1318.449489] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:27:11 (1787844431) [ 1318.760934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1318.781975] alg: No test for adler32 (adler32-zlib) [ 1319.707288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1319.798198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1319.848469] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1320.714785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1320.737993] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1322.221676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 1322.291186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1322.345382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1322.400288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1322.462816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1322.512755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1322.548271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.14@tcp [ 1322.572471] LNetError: 68187:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.14@tcp to recovery [ 1322.848886] LNet: 1 local NIs in recovery (showing 1): 192.168.203.14@tcp [ 1322.851853] LNet: There was an unexpected network error while writing to 192.168.203.14: rc = -22 [ 1324.905560] LNet: There was an unexpected network error while writing to 192.168.203.14: rc = -22 [ 1326.138125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1328.991776] LNet: There was an unexpected network error while writing to 192.168.203.14: rc = -22 [ 1329.849504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1329.876404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1329.935297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1329.960870] LNet: 68294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.965769] LNetError: 68294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1329.970337] LNet: Removed LNI 192.168.203.14@tcp [ 1329.985292] LNet: Removed LNI 192.168.203.14@tcp1 [ 1330.007389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.074535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1330.103972] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1330.798252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1330.822742] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1331.648194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 1331.699221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1331.773739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1331.842709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1331.875161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1331.927324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1331.973466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1332.013254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.14@tcp [ 1332.042474] LNetError: 68567:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.14@tcp to recovery [ 1332.063815] LNet: There was an unexpected network error while writing to 192.168.203.14: rc = -22 [ 1334.558348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1338.207928] LNet: There was an unexpected network error while writing to 192.168.203.14: rc = -22 [ 1338.218712] LNet: Skipped 1 previous similar message [ 1342.721085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1342.767868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1342.806645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.420879] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:27:43 (1787844463) [ 1349.487300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1349.514558] LNet: 69266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.527377] LNetError: 69266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.532957] LNetError: 69266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1349.540281] LNet: Removed LNI 192.168.203.14@tcp [ 1349.568583] LNet: Removed LNI 192.168.203.14@tcp1 [ 1349.615327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.671520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1349.715677] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1350.341571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1350.366422] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1351.197363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 1351.265987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1351.317464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1351.364829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1351.395553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1351.422274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1351.454715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.14@tcp --health 0 [ 1351.601550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1351.712231] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.14@tcp [ 1352.746176] LNet: There was an unexpected network error while writing to 192.168.203.14: rc = -22 [ 1352.760947] LNet: Skipped 1 previous similar message [ 1367.144890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1367.320976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1367.358630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1367.399061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.14@tcp --health 500 [ 1367.519028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1374.122604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1374.241044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1374.268852] LNet: 69765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.292352] LNetError: 69765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.303632] LNetError: 69765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1374.317787] LNet: Removed LNI 192.168.203.14@tcp [ 1374.336328] LNet: Removed LNI 192.168.203.14@tcp1 [ 1374.399736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.527438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1374.585380] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1375.262599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.280782] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1376.011711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 1376.049729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.126026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1376.169552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1376.214741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1376.242665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1376.293558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1376.333507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.14@tcp --health 0 [ 1377.509483] LNet: There was an unexpected network error while writing to 192.168.203.14: rc = -22 [ 1377.522290] LNet: Skipped 3 previous similar messages [ 1377.700799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.095718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.123145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1388.166842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1395.386699] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:28:29 (1787844509) [ 1395.975471] LNet: 70812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.979803] LNetError: 70812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.987180] LNetError: 70812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1396.001375] LNet: Removed LNI 192.168.203.14@tcp [ 1396.014974] LNet: Removed LNI 192.168.203.14@tcp1 [ 1396.756305] Key type .llcrypt unregistered [ 1396.759761] Key type ._llcrypt unregistered [ 1403.479166] Key type ._llcrypt registered [ 1403.482868] Key type .llcrypt registered [ 1403.797592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1403.844552] alg: No test for adler32 (adler32-zlib) [ 1404.856199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1404.893492] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1416.376673] Lustre: DEBUG MARKER: Initial discovery [ 1416.688215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.114@tcp [ 1420.288870] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1420.421345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1420.464823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.114@tcp [ 1420.513078] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1420.547987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1420.640784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.114@tcp -r 1 -m GET -l 3 [ 1420.674167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.114@tcp -r 1 -m GET -e local_error [ 1420.721937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.114@tcp -r 1 -m PUT -l 6 [ 1421.282844] LNet: 1 local NIs in recovery (showing 1): 192.168.203.14@tcp [ 1422.627814] Lustre: DEBUG MARKER: Force oleg314-client.virtnet to discover 192.168.203.114@tcp (in background) [ 1422.789682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.114@tcp [ 1424.450343] Lustre: DEBUG MARKER: Force oleg314-server to discover 192.168.203.14@tcp [ 1425.888972] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1426.985766] Lustre: DEBUG MARKER: Wait for 71572 [ 1428.841218] Lustre: DEBUG MARKER: Finished wait on 71572 [ 1428.944943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1428.993708] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1429.022838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1429.531799] LNet: 72033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.541948] LNetError: 72033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1429.562492] LNet: Removed LNI 192.168.203.14@tcp [ 1429.938919] Key type .llcrypt unregistered [ 1429.948097] Key type ._llcrypt unregistered [ 1447.796745] Key type ._llcrypt registered [ 1447.798683] Key type .llcrypt registered [ 1449.220222] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:29:22 (1787844562) [ 1450.109515] Key type .llcrypt unregistered [ 1450.111545] Key type ._llcrypt unregistered [ 1456.695089] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1456.978084] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1456.993812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1457.000143] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1459.687784] Key type ._llcrypt registered [ 1459.689951] Key type .llcrypt registered [ 1459.916483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1459.925399] alg: No test for adler32 (adler32-zlib) [ 1460.897373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.936457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1460.955878] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1461.696342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1461.737287] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1470.487656] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:29:44 (1787844584) [ 1470.724503] LNet: 74110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.729976] LNetError: 74110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1470.735491] LNet: Removed LNI 10.1.2.3@tcp [ 1470.747281] LNet: Removed LNI 192.168.203.14@tcp [ 1471.487725] Key type .llcrypt unregistered [ 1471.490813] Key type ._llcrypt unregistered [ 1479.102717] Key type ._llcrypt registered [ 1479.105172] Key type .llcrypt registered [ 1479.444573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1479.492552] alg: No test for adler32 (adler32-zlib) [ 1480.436574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.484324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.503095] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1481.366740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1481.384591] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1482.699614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.14@tcp [ 1483.743859] LNetError: Unexpected error -22 connecting to 192.168.203.14@tcp at host 192.168.203.14:988 [ 1491.163865] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:30:04 (1787844604) [ 1491.273454] LNet: 75237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.293454] LNetError: 75237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1491.308300] LNet: Removed LNI 10.1.2.3@tcp [ 1491.324712] LNet: Removed LNI 192.168.203.14@tcp [ 1492.063549] Key type .llcrypt unregistered [ 1492.066471] Key type ._llcrypt unregistered [ 1499.069631] Key type ._llcrypt registered [ 1499.074792] Key type .llcrypt registered [ 1499.340834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1499.356438] alg: No test for adler32 (adler32-zlib) [ 1500.356444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.446686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1500.481612] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1501.294750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1501.320266] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 1502.595798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.14@tcp1 --nid 192.168.203.14@tcp2 [ 1507.016533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.14@tcp1 192.168.203.14@tcp2 [ 1507.052089] LNetError: 76085:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.14@tcp1 to 192.168.203.14@tcp2 - Route table may be misconfigured [ 1507.086906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.14@tcp2 192.168.203.14@tcp1 [ 1513.117751] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:30:26 (1787844626) [ 1513.225658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.259759] LNet: 76652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.269227] LNetError: 76652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.278395] LNet: Removed LNI 192.168.203.14@tcp1 [ 1514.346416] LNet: Removed LNI 192.168.203.14@tcp2 [ 1514.371980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1514.423082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1514.457736] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1515.126332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1515.158523] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1515.950940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 1515.984644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.14@tcp 192.168.203.14@tcp [ 1516.019596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.14@tcp1 192.168.203.14@tcp1 [ 1516.059717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.14@tcp -d 192.168.203.14@tcp -e network_timeout [ 1516.096659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -e network_timeout [ 1516.148360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -e network_timeout [ 1516.185389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -e network_timeout [ 1516.228784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1516.248450] LNet: There was an unexpected network error while writing to 192.168.203.14: rc = -22 [ 1516.263689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1516.384449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1516.448095] LNet: 1 local NIs in recovery (showing 1): 192.168.203.14@tcp [ 1516.492519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1523.311166] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:30:36 (1787844636) [ 1523.405702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1523.426319] LNet: 77513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.437583] LNetError: 77513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.442750] LNetError: 77513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1523.449661] LNet: Removed LNI 192.168.203.14@tcp [ 1524.518667] LNet: Removed LNI 192.168.203.14@tcp1 [ 1524.553374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.639536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1524.977881] LNet: 77560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.435756] Key type .llcrypt unregistered [ 1525.437972] Key type ._llcrypt unregistered [ 1536.684746] Key type ._llcrypt registered [ 1536.687140] Key type .llcrypt registered [ 1538.120871] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:30:51 (1787844651) [ 1538.530806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1538.552984] alg: No test for adler32 (adler32-zlib) [ 1539.512493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.936334] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1543.281815] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:30:56 (1787844656) [ 1543.432249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.471300] LNet: 78652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.506314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.602816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.667798] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1544.657322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1544.677101] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1545.740200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1545.790423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp1 [ 1545.848839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp1 [ 1552.190409] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:31:05 (1787844665) [ 1553.760360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg314-server [ 1555.455318] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:31:09 (1787844669) [ 1557.172558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg314-server [ 1558.870720] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:31:12 (1787844672) [ 1560.633649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg314-server [ 1562.372501] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:31:16 (1787844676) [ 1564.005738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg314-server [ 1566.025592] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:31:19 (1787844679) [ 1567.938371] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg314-server [ 1569.542417] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:31:23 (1787844683) [ 1571.343796] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg314-server [ 1573.101959] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:31:26 (1787844686) [ 1575.194908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg314-server [ 1577.221260] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:31:30 (1787844690) [ 1579.122758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg314-server [ 1580.908161] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:31:34 (1787844694) [ 1582.832937] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg314-server [ 1584.845728] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:31:38 (1787844698) [ 1584.921299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.942180] LNet: 82530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.948037] LNetError: 82530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.954957] LNet: Removed LNI 192.168.203.14@tcp [ 1586.022872] LNet: Removed LNI 192.168.203.14@tcp1 [ 1586.049952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.113584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1586.168032] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1586.812520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1586.884718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1587.136046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.176811] LNet: 82683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.187562] LNetError: 82683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.197422] LNetError: 82683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1587.209545] LNet: Removed LNI 192.168.203.14@tcp [ 1587.237498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.306509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1587.340852] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1588.405638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1588.648142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.685769] LNetError: 82829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.699167] LNet: Removed LNI 192.168.203.14@tcp [ 1588.734964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.820529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.848371] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1589.853702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1590.047599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1590.325841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.348940] LNet: 82976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.355258] LNet: 82976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1591.409143] LNet: Removed LNI 192.168.203.14@tcp [ 1591.435196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.514989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1591.549887] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1592.432798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1592.672113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.715906] LNetError: 83121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.729214] LNetError: 83121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1592.754076] LNet: Removed LNI 192.168.203.14@tcp [ 1592.802711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.941692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.000733] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1594.018889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1594.138400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1594.423885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.452865] LNet: 83268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.458382] LNet: 83268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1595.499961] LNet: Removed LNI 192.168.203.14@tcp [ 1595.523352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.597531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1595.658119] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1596.471943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1596.797082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.843058] LNetError: 83414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.848937] LNetError: 83414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1596.863549] LNet: Removed LNI 192.168.203.14@tcp [ 1596.918061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.992281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1597.065557] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1597.948713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1598.022441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1598.267748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.333588] LNet: Removed LNI 192.168.203.14@tcp [ 1599.375747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.497514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1599.524812] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1600.192712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1600.425465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.520275] LNet: Removed LNI 192.168.203.14@tcp [ 1601.545261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.587561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.611205] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1602.178639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1602.241603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1603.527720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.544195] LNet: 83863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.548733] LNet: 83863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1604.583372] LNet: Removed LNI 192.168.203.14@tcp [ 1604.597756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.669882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1604.748610] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1605.849592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1606.238771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.299849] LNetError: 84008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.302329] LNetError: 84008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1606.323076] LNet: Removed LNI 192.168.203.14@tcp [ 1606.354711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.409756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1606.447502] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1607.337293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1607.437346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1607.804163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.871536] LNet: Removed LNI 192.168.203.14@tcp [ 1608.947649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.058848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1609.125707] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1610.265928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1611.735461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.842336] LNet: Removed LNI 192.168.203.14@tcp [ 1612.864052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.936410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.970424] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1613.995462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1614.053764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.080416] LNet: Removed LNI 192.168.203.14@tcp [ 1614.101931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.182192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.200299] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1615.130447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1615.212730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.270761] LNet: Removed LNI 192.168.203.14@tcp [ 1615.312359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.379466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1615.476325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.549616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.647039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1615.749386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.793182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.862807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1615.976837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.050284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.138879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1616.164475] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1617.333795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.417974] LNet: Removed LNI 192.168.203.14@tcp [ 1617.437150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.580406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1625.564201] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:32:18 (1787844738) [ 1625.771414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.818533] LNet: 85330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.831515] LNet: 85330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1625.876806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.971221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1626.001359] LNet: Added LNI 192.168.203.14@tcp231 [8/256/0/180] [ 1627.736361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.776691] LNetError: 85471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.785313] LNetError: 85471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1627.799843] LNet: Removed LNI 192.168.203.14@tcp231 [ 1627.831190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.900756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1627.931531] LNet: Added LNI 192.168.203.14@tcp231 [8/256/0/180] [ 1627.947508] LNet: 85479:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1628.488492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1628.555073] LNet: Removed LNI 192.168.203.14@tcp231 [ 1628.582987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1628.637582] LNet: Added LNI 192.168.203.14@tcp231 [8/256/0/180] [ 1629.996247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.030672] LNet: Removed LNI 192.168.203.14@tcp231 [ 1630.046333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.132464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1630.157379] LNet: Added LNI 192.168.203.14@tcp231 [8/256/0/0] [ 1630.165808] LNet: 85632:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1630.808196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1630.857741] LNet: Removed LNI 192.168.203.14@tcp231 [ 1630.872091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1630.899527] LNet: Added LNI 192.168.203.14@tcp231 [8/256/0/0] [ 1632.071246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.111578] LNet: Removed LNI 192.168.203.14@tcp231 [ 1632.140038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.206428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1632.224797] LNet: Added LNI 192.168.203.14@tcp231 [8/256/0/60] [ 1632.233104] LNet: 85783:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1632.722182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1632.757463] LNet: Removed LNI 192.168.203.14@tcp231 [ 1632.795459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1632.854452] LNet: Added LNI 192.168.203.14@tcp231 [8/256/0/60] [ 1633.984950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.025226] LNet: Removed LNI 192.168.203.14@tcp231 [ 1634.047129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.139658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1634.159156] LNet: Added LNI 192.168.203.14@tcp231 [8/256/0/180] [ 1634.853573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1634.900176] LNet: Removed LNI 192.168.203.14@tcp231 [ 1634.921461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1634.954556] LNet: Added LNI 192.168.203.14@tcp231 [8/256/0/180] [ 1643.551208] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:32:37 (1787844757) [ 1643.686271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.737538] LNet: Removed LNI 192.168.203.14@tcp231 [ 1643.781140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.923867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1643.977233] LNet: Added LNI 192.168.203.14@tcp232 [8/256/0/180] [ 1644.977456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1645.043464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1645.071660] LNet: Removed LNI 192.168.203.14@tcp232 [ 1645.096881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1645.134419] LNet: Added LNI 192.168.203.14@tcp232 [8/256/0/180] [ 1646.243740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1646.298973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1646.342755] LNet: Removed LNI 192.168.203.14@tcp232 [ 1646.365437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1646.393640] LNet: Added LNI 192.168.203.14@tcp232 [8/256/0/180] [ 1647.283626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1647.348818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1647.391720] LNet: Removed LNI 192.168.203.14@tcp232 [ 1647.431911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1647.506634] LNet: Added LNI 192.168.203.14@tcp232 [8/256/0/180] [ 1648.355877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1648.417384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1648.461835] LNet: Removed LNI 192.168.203.14@tcp232 [ 1648.489534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1654.439394] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:32:48 (1787844768) [ 1655.828602] Key type .llcrypt unregistered [ 1655.837701] Key type ._llcrypt unregistered [ 1662.710608] Key type ._llcrypt registered [ 1662.714961] Key type .llcrypt registered [ 1662.894915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1662.912861] alg: No test for adler32 (adler32-zlib) [ 1663.872114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1663.900582] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1671.516960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 1681.463115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.528032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1681.560924] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1683.119473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_interrupt -r 2 [ 1683.184643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_interrupt -r 2 [ 1683.235621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_interrupt -r 2 [ 1683.285551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_interrupt -r 2 [ 1683.335506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_interrupt -r 2 [ 1683.420722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_interrupt -r 2 [ 1683.511604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_interrupt -r 2 [ 1683.580741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_interrupt -r 2 [ 1683.633892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.14@tcp 192.168.203.114@tcp [ 1683.673553] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1683.688715] LNet: 1 local NIs in recovery (showing 1): 192.168.203.14@tcp [ 1684.058434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1684.107518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1684.156718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1684.201795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1684.243805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1684.283906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1684.314620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1684.356080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1684.372893] LNet: 88442:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1684.394758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1684.424817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1684.453289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1684.480512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1684.528855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1684.596863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1685.253710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_dropped -r 2 [ 1685.351842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_dropped -r 2 [ 1685.424373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_dropped -r 2 [ 1685.506952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_dropped -r 2 [ 1685.587366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_dropped -r 2 [ 1685.624742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_dropped -r 2 [ 1685.669847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_dropped -r 2 [ 1685.732571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_dropped -r 2 [ 1685.778802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.14@tcp 192.168.203.114@tcp [ 1685.817894] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1686.099037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1686.140589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1686.173580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1686.196730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1686.221029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1686.251919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1686.295907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1686.334114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1686.353969] LNet: 88562:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1686.380263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1686.418135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1686.454457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1686.510444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1686.557188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1686.602908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1687.142530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_aborted -r 2 [ 1687.175794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_aborted -r 2 [ 1687.203640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_aborted -r 2 [ 1687.231570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_aborted -r 2 [ 1687.259291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_aborted -r 2 [ 1687.309163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_aborted -r 2 [ 1687.354073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_aborted -r 2 [ 1687.388543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_aborted -r 2 [ 1687.426334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.14@tcp 192.168.203.114@tcp [ 1687.442401] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1687.705325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1687.733779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1687.763832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1687.789541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1687.816778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1687.848428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1687.900212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1687.935953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1687.961888] LNet: 88683:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1687.983325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1688.025633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1688.050631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1688.078562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1688.110371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1688.143672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1688.663395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_no_route -r 2 [ 1688.701803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_no_route -r 2 [ 1688.739980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_no_route -r 2 [ 1688.786374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_no_route -r 2 [ 1688.822640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_no_route -r 2 [ 1688.850096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_no_route -r 2 [ 1688.881548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_no_route -r 2 [ 1688.912145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_no_route -r 2 [ 1688.950637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.14@tcp 192.168.203.114@tcp [ 1689.309460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1689.368783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1689.407381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1689.473514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1689.518055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1689.583275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1689.630684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1689.665895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1689.713838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1689.772920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1689.808694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1689.840230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1689.868800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1689.915648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1690.393610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e local_timeout -r 2 [ 1690.433927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e local_timeout -r 2 [ 1690.468765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e local_timeout -r 2 [ 1690.515303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e local_timeout -r 2 [ 1690.558719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e local_timeout -r 2 [ 1690.599847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e local_timeout -r 2 [ 1690.657833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e local_timeout -r 2 [ 1690.700697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e local_timeout -r 2 [ 1690.745313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.14@tcp 192.168.203.114@tcp [ 1690.766725] LNet: There was an unexpected network error while writing to 192.168.203.114: rc = -22 [ 1690.775399] LNet: Skipped 1 previous similar message [ 1690.970810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1690.989854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1691.012778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1691.047866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1691.083974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1691.118808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1691.148582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1691.178426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1691.191746] LNet: 88923:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1691.195757] LNet: 88923:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1691.217272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1691.252499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1691.293558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1691.319745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1691.377667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1691.401971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1691.817443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp -m GET -e remote_dropped -r 2 [ 1691.852321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.114@tcp1 -m GET -e remote_dropped -r 2 [ 1691.888282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp -m GET -e remote_dropped -r 2 [ 1691.922292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp -d 192.168.203.14@tcp1 -m GET -e remote_dropped -r 2 [ 1691.967390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp -m GET -e remote_dropped -r 2 [ 1692.016680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.114@tcp1 -m GET -e remote_dropped -r 2 [ 1692.040332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp -m GET -e remote_dropped -r 2 [ 1692.063667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.14@tcp1 -d 192.168.203.14@tcp1 -m GET -e remote_dropped -r 2 [ 1692.092963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.14@tcp 192.168.203.114@tcp [ 1692.121641] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.114@tcp [ 1692.375127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1692.409272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1692.435238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1692.458625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1692.488878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1692.519978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1692.561454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1692.593565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1692.617391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1692.648325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1692.678073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1692.708442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1692.735652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1692.771461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1699.098958] LNet: 89121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.108861] LNetError: 89121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.117926] LNet: Removed LNI 192.168.203.14@tcp [ 1699.126152] LNet: Removed LNI 192.168.203.14@tcp1 [ 1699.471111] Key type .llcrypt unregistered [ 1699.472975] Key type ._llcrypt unregistered [ 1719.420553] Key type ._llcrypt registered [ 1719.423235] Key type .llcrypt registered [ 1721.251415] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:33:54 (1787844834) [ 1721.579411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1721.592871] alg: No test for adler32 (adler32-zlib) [ 1722.540661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.605643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.632421] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1723.305565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1723.352370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.14@tcp [ 1723.376314] LNetError: 90102:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.14@tcp to recovery [ 1723.460237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.14@tcp [ 1723.563178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1729.332862] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:34:03 (1787844843) [ 1729.424192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.449481] LNet: 90684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.461894] LNetError: 90684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.510342] LNet: Removed LNI 192.168.203.14@tcp [ 1729.537958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.641900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.665512] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1730.278484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1730.309903] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 1731.335940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.14@tcp [ 1731.362921] LNetError: 90960:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.14@tcp to recovery [ 1731.679355] LNet: 1 local NIs in recovery (showing 1): 192.168.203.14@tcp [ 1731.684978] LNet: There was an unexpected network error while writing to 192.168.203.14: rc = -22 [ 1732.562484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.14@tcp [ 1741.101455] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:34:14 (1787844854) [ 1743.184404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg314-server [ 1744.750846] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:34:18 (1787844858) [ 1747.046715] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg314-server [ 1749.360464] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:34:22 (1787844862) [ 1751.646849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg314-server [ 1753.368303] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:34:27 (1787844867) [ 1753.723324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.776590] LNet: 92581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.787121] LNetError: 92581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.805726] LNet: Removed LNI 192.168.203.14@tcp [ 1753.831194] LNet: Removed LNI 192.168.203.14@tcp1 [ 1753.898277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.010738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.063766] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1754.865929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1754.934304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1754.973481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1755.038559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1755.081682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1755.124627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1761.456066] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:34:35 (1787844875) [ 1761.572001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.606733] LNet: 93299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.621680] LNetError: 93299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.631227] LNetError: 93299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1761.644966] LNet: Removed LNI 192.168.203.14@tcp [ 1761.667129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.758212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.782779] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1768.109393] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:34:41 (1787844881) [ 1768.236765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.268366] LNet: 94005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.286777] LNetError: 94005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.301424] LNet: Removed LNI 192.168.203.14@tcp [ 1768.337198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.408488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1768.434651] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1774.723494] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:34:48 (1787844888) [ 1776.229379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1777.763801] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:34:51 (1787844891) [ 1778.367776] LNet: 95044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.378221] LNetError: 95044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.391403] LNet: Removed LNI 192.168.203.14@tcp [ 1778.901301] Key type .llcrypt unregistered [ 1778.904971] Key type ._llcrypt unregistered [ 1784.781567] Key type ._llcrypt registered [ 1784.786942] Key type .llcrypt registered [ 1785.019418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1785.046625] alg: No test for adler32 (adler32-zlib) [ 1785.977585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1786.012879] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1793.403686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 1797.456660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.114@tcp [ 1797.796357] LNet: 95469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.804559] LNetError: 95469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.823503] LNet: Removed LNI 192.168.203.14@tcp [ 1798.134624] Key type .llcrypt unregistered [ 1798.137742] Key type ._llcrypt unregistered [ 1809.862254] Key type ._llcrypt registered [ 1809.867670] Key type .llcrypt registered [ 1811.465805] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1813.098602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1814.722707] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:35:28 (1787844928) [ 1816.520557] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1818.127287] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:35:31 (1787844931) [ 1820.100297] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg314-server [ 1822.087128] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:35:35 (1787844935) [ 1824.019340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg314-server [ 1826.083300] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:35:39 (1787844939) [ 1827.822458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1827.858653] alg: No test for adler32 (adler32-zlib) [ 1828.998826] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1829.253768] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1830.967154] Key type lgssc registered [ 1831.970583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1852.417160] Key type lgssc unregistered [ 1852.725589] LNet: 98647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.732894] LNetError: 98647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.747702] LNet: Removed LNI 192.168.203.14@tcp [ 1853.630198] Key type .llcrypt unregistered [ 1853.631781] Key type ._llcrypt unregistered [ 1871.140984] Key type ._llcrypt registered [ 1871.146223] Key type .llcrypt registered [ 1872.348744] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:36:25 (1787844985) [ 1872.773786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1872.790182] alg: No test for adler32 (adler32-zlib) [ 1873.752803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.810239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1873.857266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1873.901339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1873.946869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1880.496148] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:36:34 (1787844994) [ 1880.821261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.852713] LNet: 100065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.881197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.987513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1881.031112] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1881.987566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1882.005620] LNet: 100209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.010980] LNetError: 100209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.024286] LNet: Removed LNI 192.168.203.14@tcp [ 1882.040523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.092533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1882.113417] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1882.736600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1882.782331] LNetError: 100237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.792219] LNet: Removed LNI 192.168.203.14@tcp [ 1882.811104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.878433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1882.901603] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1890.231578] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:36:43 (1787845003) [ 1890.484679] LNet: 100827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.494688] LNet: 100827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1890.507353] LNetError: 100827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.527361] LNet: Removed LNI 192.168.203.14@tcp [ 1891.405448] Key type .llcrypt unregistered [ 1891.408966] Key type ._llcrypt unregistered [ 1897.282567] Key type ._llcrypt registered [ 1897.286080] Key type .llcrypt registered [ 1897.524794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1897.546835] alg: No test for adler32 (adler32-zlib) [ 1898.592260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.726505] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1898.730748] LNet: *** cfs_fail_loc=e003, val=100*** [ 1899.767904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1899.790364] LNet: *** cfs_fail_loc=e003, val=100*** [ 1899.807352] LNet: Skipped 2 previous similar messages [ 1899.817994] Ping reply received with mapping: [ 1899.821628] found metadata reply NI[192.168.203.14@tcp] [ 1899.824956] 0e cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1899.832835] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1899.841520] 5a 5a 5a 5a 5a 5a 5a 5a [ 1900.063820] LNet: 101229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.070186] LNetError: 101229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.111033] LNet: Removed LNI 192.168.203.14@tcp [ 1900.925346] Key type .llcrypt unregistered [ 1900.931190] Key type ._llcrypt unregistered [ 1912.904739] Key type ._llcrypt registered [ 1912.909248] Key type .llcrypt registered [ 1914.796588] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:37:07 (1787845027) [ 1915.564643] Key type .llcrypt unregistered [ 1915.569577] Key type ._llcrypt unregistered [ 1922.364360] Key type ._llcrypt registered [ 1922.368042] Key type .llcrypt registered [ 1922.576453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1922.601254] alg: No test for adler32 (adler32-zlib) [ 1923.786628] LNetError: 102287:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1924.167613] Key type .llcrypt unregistered [ 1924.169072] Key type ._llcrypt unregistered [ 1929.747476] Key type ._llcrypt registered [ 1929.751265] Key type .llcrypt registered [ 1931.343777] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:37:24 (1787845044) [ 1931.917559] Key type .llcrypt unregistered [ 1931.919454] Key type ._llcrypt unregistered [ 1936.885845] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1936.900953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1939.488894] Key type ._llcrypt registered [ 1939.490431] Key type .llcrypt registered [ 1939.645261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1939.665987] alg: No test for adler32 (adler32-zlib) [ 1940.589731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1940.655132] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1949.755191] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:37:43 (1787845063) [ 1949.896764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.938406] LNet: 104076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.960547] LNetError: 104076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1949.984324] LNet: Removed LNI 10.1.2.3@tcp [ 1949.998780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.770429] Key type .llcrypt unregistered [ 1950.772685] Key type ._llcrypt unregistered [ 1956.207535] Key type ._llcrypt registered [ 1956.214996] Key type .llcrypt registered [ 1956.451071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1956.471987] alg: No test for adler32 (adler32-zlib) [ 1957.424433] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1957.516144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 1957.720754] LNet: 104370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.728551] LNetError: 104370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.759577] LNet: Removed LNI 192.168.203.14@tcp [ 1959.371563] Key type .llcrypt unregistered [ 1959.374567] Key type ._llcrypt unregistered [ 1972.521623] Key type ._llcrypt registered [ 1972.524547] Key type .llcrypt registered [ 1973.994359] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:38:07 (1787845087) [ 1975.034458] Key type .llcrypt unregistered [ 1975.044662] Key type ._llcrypt unregistered [ 1981.253621] Key type ._llcrypt registered [ 1981.258800] Key type .llcrypt registered [ 1981.456249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1981.471352] alg: No test for adler32 (adler32-zlib) [ 1988.547975] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:38:22 (1787845102) [ 1988.680482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1988.723285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.836350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1988.873675] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 1989.664347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1989.701910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1989.758820] LNet: Removed LNI 192.168.203.14@tcp [ 1990.169920] LNet: 106126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.617733] Key type .llcrypt unregistered [ 1990.623797] Key type ._llcrypt unregistered [ 2001.113654] Key type ._llcrypt registered [ 2001.115582] Key type .llcrypt registered [ 2002.158670] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:38:36 (1787845116) [ 2002.558561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2002.566779] alg: No test for adler32 (adler32-zlib) [ 2003.531141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.578584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2003.600921] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2004.540360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2004.614180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.14@tcp [ 2004.653333] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2004.663636] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2004.669931] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2004, 0, 0 [ 2004.675770] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2004.678249] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2004.686407] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2004.694364] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2004, 0, 0 [ 2004.704539] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2004.710165] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2004.717549] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2004.723091] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2004, 0, 2054 [ 2004.729928] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2004.732993] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2004.737249] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2004.740926] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2004, 0, 2054 [ 2004.748518] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2011.178452] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:38:44 (1787845124) [ 2011.754819] LNet: 107665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.760370] LNetError: 107665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.772722] LNet: Removed LNI 192.168.203.14@tcp [ 2012.340443] Key type .llcrypt unregistered [ 2012.342666] Key type ._llcrypt unregistered [ 2017.728339] Key type ._llcrypt registered [ 2017.732886] Key type .llcrypt registered [ 2017.897860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2017.912677] alg: No test for adler32 (adler32-zlib) [ 2018.974399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2019.022784] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2026.335305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 2036.714844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.764746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2036.789870] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 2037.737811] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2037.831485] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2037.837131] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2037.841329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2040.181337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2040.207622] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2050.942760] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:39:24 (1787845164) [ 2051.109622] LNet: 109167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.124522] LNetError: 109167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.131191] LNet: Removed LNI 192.168.203.14@tcp [ 2052.191192] LNet: Removed LNI 192.168.203.14@tcp1 [ 2052.197224] LNetError: 109167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2052.213812] LNetError: 109167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2052.271964] LNet: Removed LNI 10.1.2.3@tcp99 [ 2053.065471] Key type .llcrypt unregistered [ 2053.069899] Key type ._llcrypt unregistered [ 2058.197262] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2058.203819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2060.596941] Key type ._llcrypt registered [ 2060.599669] Key type .llcrypt registered [ 2060.742373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2060.761111] alg: No test for adler32 (adler32-zlib) [ 2061.674570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.751634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2061.777207] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2062.303219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2062.319370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2064.069526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2064.113840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2064.165398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2064.186958] LNetError: 109925:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2064.221254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2064.258988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2064.860787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2064.928184] LNet: 109945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.932546] LNetError: 109945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2064.937330] LNet: Removed LNI 10.1.2.3@tcp [ 2065.960145] LNet: Removed LNI 192.168.203.14@tcp [ 2066.584437] Key type .llcrypt unregistered [ 2066.589425] Key type ._llcrypt unregistered [ 2076.943613] Key type ._llcrypt registered [ 2076.945027] Key type .llcrypt registered [ 2078.427121] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:39:51 (1787845191) [ 2078.800061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2078.835104] alg: No test for adler32 (adler32-zlib) [ 2079.765285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.834615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2079.861746] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2080.768658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2080.908402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg314-client@tcp [ 2088.209286] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:40:01 (1787845201) [ 2088.324784] LNet: 111477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.333108] LNetError: 111477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.347656] LNet: Removed LNI 192.168.203.14@tcp [ 2089.285021] Key type .llcrypt unregistered [ 2089.287931] Key type ._llcrypt unregistered [ 2097.917884] Key type ._llcrypt registered [ 2097.924193] Key type .llcrypt registered [ 2098.169053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2098.196132] alg: No test for adler32 (adler32-zlib) [ 2099.159878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2099.184558] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2099.230336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2109.262981] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:40:22 (1787845222) [ 2111.325745] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2113.753963] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:40:26 (1787845226) [ 2115.416362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2117.435883] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:40:30 (1787845230) [ 2117.648208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.712763] LNet: 112908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.733581] LNetError: 112908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.754341] LNet: Removed LNI 192.168.203.14@tcp [ 2117.790700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.847163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2117.890134] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2118.869628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2118.929360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2126.275578] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:40:39 (1787845239) [ 2126.433648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.468737] LNet: 113616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.481817] LNetError: 113616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.507485] LNet: Removed LNI 192.168.203.14@tcp [ 2126.543126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.623402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2126.711940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2127.495519] Key type .llcrypt unregistered [ 2127.505276] Key type ._llcrypt unregistered [ 2139.819969] Key type ._llcrypt registered [ 2139.826107] Key type .llcrypt registered [ 2141.080591] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:40:54 (1787845254) [ 2141.594692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2141.609390] alg: No test for adler32 (adler32-zlib) [ 2142.554181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.606338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2142.628685] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2143.400345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2143.502501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2149.454738] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:41:03 (1787845263) [ 2149.549055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.585411] LNet: 115170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.600324] LNetError: 115170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2149.622086] LNet: Removed LNI 192.168.203.14@tcp [ 2149.659825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.761641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2149.806498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2156.223677] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:41:09 (1787845269) [ 2156.329623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.365060] LNet: 115751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.401123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.506936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2156.984838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2157.054146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2157.170232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2157.592355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2157.631123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2157.737778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.756706] LNet: 115791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.780745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.884923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2158.321993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2158.361272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2158.434721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2158.817261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2158.867992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2165.426692] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:41:18 (1787845278) [ 2165.545367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.595288] LNet: 116387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.624874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.723052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.746239] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2166.670762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2166.711549] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2167.829994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2167.884425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2168.058709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.075206] LNetError: 116670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.081736] LNet: Removed LNI 192.168.203.14@tcp [ 2168.092734] LNet: Removed LNI 192.168.203.14@tcp2 [ 2168.112112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.174079] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2168.183356] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2168.190685] LNet: 116676:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2168.678515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2168.856681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2169.887249] LNet: Removed LNI 192.168.203.14@tcp [ 2169.889941] LNetError: 116714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.898631] LNetError: 116714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2169.943314] LNet: Removed LNI 192.168.203.14@tcp2 [ 2169.984388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.053211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2170.080233] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2170.804622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2170.837926] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2171.664794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2171.745651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2171.844652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2172.113065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2172.143211] LNet: 116998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.151022] LNet: 116998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2172.156890] LNetError: 116998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2173.218120] LNet: Removed LNI 192.168.203.14@tcp [ 2174.247262] LNet: Removed LNI 192.168.203.14@tcp2 [ 2174.298936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2174.419399] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2174.432184] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2174.440198] LNet: 117004:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2174.912207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2175.100434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.135887] LNet: Removed LNI 192.168.203.14@tcp [ 2176.171541] LNet: Removed LNI 192.168.203.14@tcp2 [ 2176.202931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2176.271964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2176.304537] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2177.026752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2177.053385] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2177.951742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2177.996781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2178.209953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2178.230303] LNetError: 117321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2178.240634] LNetError: 117321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2178.250861] LNet: Removed LNI 192.168.203.14@tcp [ 2178.271818] LNet: Removed LNI 192.168.203.14@tcp2 [ 2178.306154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.406275] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2178.413942] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2178.425716] LNet: 117327:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2178.908607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2179.167881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.191187] LNet: Removed LNI 192.168.203.14@tcp [ 2180.207241] LNet: Removed LNI 192.168.203.14@tcp2 [ 2180.225643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.315617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.351603] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2181.069621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2181.097001] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2181.920763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2181.967513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2182.006647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2182.243214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.262342] LNet: 117649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.268260] LNet: 117649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2183.327305] LNet: Removed LNI 192.168.203.14@tcp [ 2184.422174] LNet: Removed LNI 192.168.203.14@tcp2 [ 2184.478992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.576457] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2184.584284] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2184.591059] LNet: 117655:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2185.308865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2185.669135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2185.690464] LNet: Removed LNI 192.168.203.14@tcp [ 2186.738170] LNet: Removed LNI 192.168.203.14@tcp2 [ 2186.808399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.902367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2186.936238] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2187.763825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2187.796406] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2188.781215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2188.804948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2188.866120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2189.201368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.229213] LNetError: 117989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.239544] LNetError: 117989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2189.248359] LNet: Removed LNI 192.168.203.14@tcp [ 2189.259205] LNet: Removed LNI 192.168.203.14@tcp2 [ 2189.302086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.387573] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2189.394550] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2189.398807] LNet: 117995:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2190.060947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2190.555152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.647152] LNet: Removed LNI 192.168.203.14@tcp [ 2191.659182] LNet: Removed LNI 192.168.203.14@tcp2 [ 2191.707635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.826262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2191.861330] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2192.650641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2192.688215] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2193.782433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2193.818712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2193.853417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2193.923776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2194.290276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2195.359660] LNet: Removed LNI 192.168.203.14@tcp [ 2196.394412] LNet: Removed LNI 192.168.203.14@tcp2 [ 2196.474342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2196.546997] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2196.553894] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2196.917220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2204.745252] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2206.645850] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:41:59 (1787845319) [ 2206.733491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2206.769752] LNet: 119112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.776840] LNet: 119112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2206.782920] LNetError: 119112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.788718] LNetError: 119112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2206.795802] LNet: Removed LNI 192.168.203.14@tcp [ 2206.823908] LNet: Removed LNI 192.168.203.14@tcp2 [ 2206.843344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.163653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2207.201541] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2207.854538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2207.870736] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2208.912434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.932707] LNet: Removed LNI 10.1.2.3@tcp [ 2208.945155] LNet: Removed LNI 192.168.203.14@tcp [ 2208.961129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.036803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2209.086212] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2209.094514] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2209.668713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2209.703568] LNet: Removed LNI 192.168.203.14@tcp [ 2209.724585] LNet: Removed LNI 10.1.2.3@tcp [ 2216.409755] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 11:42:10 (1787845330) [ 2216.584876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.674021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.765905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2216.811055] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2217.772837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2217.818688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2217.882793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2217.922016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2217.984633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2218.033961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2218.081590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2218.169623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2218.278873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2218.335624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2218.416894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.14@tcp [ 2218.699485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2226.278656] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:42:19 (1787845339) [ 2226.450730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.560104] LNet: Removed LNI 192.168.203.14@tcp [ 2227.598489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.736870] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2227.931723] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2227.954374] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2227.969336] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2230.499291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2230.559583] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2231.576676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2231.595636] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2234.355521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2234.433709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2234.508648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.14@tcp -m PUT -e local_timeout -r 1 [ 2234.569906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2234.607279] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2234.610511] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2234.891878] LNetError: 120798:0:(lib-move.c:4801:lnet_parse()) 192.168.203.14@tcp, src 192.168.203.14@tcp: Dropping PUT (error -108 looking up sender) [ 2236.961649] LNet: Removed LNI 10.1.2.3@tcp [ 2237.992551] LNet: Removed LNI 192.168.203.14@tcp [ 2238.398425] Key type .llcrypt unregistered [ 2238.403079] Key type ._llcrypt unregistered [ 2251.084327] Key type ._llcrypt registered [ 2251.086799] Key type .llcrypt registered [ 2252.319834] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:42:46 (1787845366) [ 2253.040615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2253.056696] alg: No test for adler32 (adler32-zlib) [ 2254.035137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.447655] LNet: 122115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.018190] Key type .llcrypt unregistered [ 2255.021132] Key type ._llcrypt unregistered [ 2261.147683] Key type ._llcrypt registered [ 2261.150690] Key type .llcrypt registered [ 2261.333275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2261.349216] alg: No test for adler32 (adler32-zlib) [ 2262.229913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2262.264315] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2268.223923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 2268.383653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2268.772628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.114@tcp [ 2268.987549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2269.014566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2269.035368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2269.055645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2269.073715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2269.092666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2269.112564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2269.136612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2269.151199] LNet: 122569:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2269.163120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2269.181905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2269.206517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2269.225125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2269.244778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2269.269250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2269.797832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2269.962679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2269.987351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2270.014975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2270.041528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2270.065464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2270.089624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2270.122580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2270.153375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2270.171728] LNet: 122686:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2270.186722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2270.222230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2270.260264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2270.287517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2270.320881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2270.347254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2270.766892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.114@tcp [ 2270.967431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2270.999572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2271.026254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2271.056232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2271.083211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2271.133290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2271.178840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2271.221765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2271.243706] LNet: 122803:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2271.270278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2271.313593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2271.340561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2271.358245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2271.376710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2271.400184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2271.872089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2272.073800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2272.110834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2272.145739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2272.177947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2272.217217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2272.242636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2272.280636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2272.317416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2272.341670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2272.369693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2272.396342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2272.427701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2272.454300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2272.487214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2272.749488] LNet: 122970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.754773] LNetError: 122970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.765760] LNet: Removed LNI 192.168.203.14@tcp [ 2273.076769] Key type .llcrypt unregistered [ 2273.081225] Key type ._llcrypt unregistered [ 2285.156265] Key type ._llcrypt registered [ 2285.159361] Key type .llcrypt registered [ 2286.411928] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:43:20 (1787845400) [ 2286.745929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2286.762931] alg: No test for adler32 (adler32-zlib) [ 2287.646571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.941186] LNet: 123858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.464787] Key type .llcrypt unregistered [ 2288.469484] Key type ._llcrypt unregistered [ 2293.518786] Key type ._llcrypt registered [ 2293.521845] Key type .llcrypt registered [ 2293.754772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2293.779321] alg: No test for adler32 (adler32-zlib) [ 2294.726300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2294.752523] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2301.377385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.114@tcp [ 2310.420753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.531838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2310.567923] LNet: Added LNI 192.168.203.14@tcp1 [8/256/0/180] [ 2311.749536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2312.328958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.114@tcp [ 2312.549548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2312.574954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2312.632324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2312.687273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2312.767425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2312.832702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2312.927620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2313.005276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2313.041838] LNet: 124515:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2313.078445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2313.131371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2313.175419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2313.183752] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.114@tcp [ 2313.218488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2313.267175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2313.333089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2314.161297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2314.471419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2314.497965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2314.525175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2314.552461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2314.577484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2314.606936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2314.638984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2314.665916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2314.681031] LNet: 124633:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2314.699938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2314.729502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2314.754376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2314.785433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2314.821353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2314.848308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2315.425640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.114@tcp [ 2315.580720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2315.602071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2315.622742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2315.650354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2315.704268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2315.748912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2315.796712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2315.855933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2315.890940] LNet: 124750:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2315.918926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2315.957181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2315.995476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2316.036244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2316.079521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2316.115448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2317.020684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2317.376165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2317.435264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2317.509197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2317.558028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2317.620367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2317.682993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2317.747515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2317.817308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2317.848447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2317.894355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2317.935761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2317.973430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2318.033738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2318.080754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2323.097305] LNet: 124952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.103169] LNetError: 124952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.110468] LNet: Removed LNI 192.168.203.14@tcp [ 2323.117917] LNet: Removed LNI 192.168.203.14@tcp1 [ 2323.518495] Key type .llcrypt unregistered [ 2323.522518] Key type ._llcrypt unregistered [ 2338.427108] Key type ._llcrypt registered [ 2338.430948] Key type .llcrypt registered [ 2339.741891] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 11:44:13 (1787845453) [ 2340.131640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2340.150814] alg: No test for adler32 (adler32-zlib) [ 2341.023443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.062622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2341.078978] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2341.798541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2341.821258] LNet: Added LNI 192.168.203.14@tcp2 [8/256/0/180] [ 2342.995861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.14@tcp [ 2343.166573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2344.223228] LNet: Removed LNI 192.168.203.14@tcp [ 2353.957796] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 11:44:27 (1787845467) [ 2355.797529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2357.415464] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:44:31 (1787845471) [ 2364.940602] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:44:38 (1787845478) [ 2372.498222] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:44:46 (1787845486) [ 2378.904410] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:44:52 (1787845492) [ 2385.454959] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:44:59 (1787845499) [ 2393.154404] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:45:06 (1787845506) [ 2403.720447] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:45:16 (1787845516) [ 2413.815497] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:45:26 (1787845526) [ 2420.134690] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:45:33 (1787845533) [ 2427.223519] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:45:40 (1787845540) [ 2434.057967] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:45:47 (1787845547) [ 2441.370545] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:45:55 (1787845555) [ 2449.741518] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:46:02 (1787845562) [ 2456.953946] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:46:10 (1787845570) [ 2464.672644] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:46:18 (1787845578) [ 2473.705855] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:46:26 (1787845586) [ 2481.405649] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:46:35 (1787845595) [ 2488.357397] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:46:42 (1787845602) [ 2488.807569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2488.843789] LNet: 136649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.857450] LNetError: 136649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2488.882657] LNet: Removed LNI 192.168.203.14@tcp2 [ 2488.922642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2489.057216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2489.107290] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2490.425732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2490.450377] LNet: 136791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2490.464303] LNetError: 136791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2490.495925] LNet: Removed LNI 192.168.203.14@tcp [ 2490.525638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2490.597289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2490.631769] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2498.295568] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:46:51 (1787845611) [ 2506.720231] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:46:59 (1787845619) [ 2514.580421] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:47:08 (1787845628) [ 2523.099699] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:47:16 (1787845636) [ 2530.657066] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:47:24 (1787845644) [ 2540.847979] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:47:33 (1787845653) [ 2548.477650] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:47:41 (1787845661) [ 2557.143673] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:47:50 (1787845670) [ 2565.166450] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:47:58 (1787845678) [ 2572.857231] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:48:06 (1787845686) [ 2580.163757] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:48:13 (1787845693) [ 2587.403957] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:48:20 (1787845700) [ 2594.543531] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2340 sec ========== 11:48:28 (1787845708) [ 2594.977265] LNet: 144261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2594.998707] LNetError: 144261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2595.018733] LNet: Removed LNI 192.168.203.14@tcp [ 2595.803623] Key type .llcrypt unregistered [ 2595.806527] Key type ._llcrypt unregistered [ 2609.735376] Key type ._llcrypt registered [ 2609.737060] Key type .llcrypt registered [ 2610.214103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2610.243670] alg: No test for adler32 (adler32-zlib) [ 2611.283588] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2611.606107] LNet: Added LNI 192.168.203.14@tcp [8/256/0/180] [ 2613.319210] Key type lgssc registered [ 2614.496685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2685.482372] Lustre: Mounted lustre-client - version 2.17.57_89_g2dd35b7 [ 2690.225394] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2698.222096] Lustre: Unmounted lustre-client [ 2724.740508] Key type lgssc unregistered [ 2724.981818] LNet: 147222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2724.990131] LNetError: 147222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2725.037421] LNet: Removed LNI 192.168.203.14@tcp [ 2725.830552] Key type .llcrypt unregistered [ 2725.833581] Key type ._llcrypt unregistered