[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 436767477 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003331] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008769] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011141] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013030] SELinux: Initializing. [ 0.014147] *** VALIDATE selinux *** [ 0.023091] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027746] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028219] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031036] *** VALIDATE tmpfs *** [ 0.033004] *** VALIDATE proc *** [ 0.034254] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040000] Speculative Store Bypass: Vulnerable [ 0.041000] debug: unmapping init [mem 0xffffffffb9859000-0xffffffffb9860fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043748] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044020] ... version: 2 [ 0.044978] ... bit width: 48 [ 0.045012] ... generic registers: 4 [ 0.046010] ... value mask: 0000ffffffffffff [ 0.047015] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049014] ... event mask: 000000070000000f [ 0.050331] rcu: Hierarchical SRCU implementation. [ 0.052354] smp: Bringing up secondary CPUs ... [ 0.053504] x86: Booting SMP configuration: [ 0.054018] .... node #0, CPUs: #1 #2 #3 [ 0.064123] smp: Brought up 1 node, 4 CPUs [ 0.066022] smpboot: Max logical packages: 1 [ 0.067027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132971] node 0 deferred pages initialised in 63ms [ 0.135048] devtmpfs: initialized [ 0.136357] x86/mm: Memory block size: 128MB [ 0.139578] gcov: version magic: 0x41383552 [ 0.141429] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142205] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143321] pinctrl core: initialized pinctrl subsystem [ 0.144275] [ 0.144958] ************************************************************* [ 0.145016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146020] ** ** [ 0.147017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.148017] ** ** [ 0.149017] ** This means that this kernel is built to expose internal ** [ 0.150014] ** IOMMU data structures, which may compromise security on ** [ 0.151014] ** your system. ** [ 0.152015] ** ** [ 0.153013] ** If you see this message and you are not debugging the ** [ 0.154014] ** kernel, report this immediately to your vendor! ** [ 0.155014] ** ** [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157012] ************************************************************* [ 0.158941] NET: Registered protocol family 16 [ 0.159465] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.160062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.161070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.162708] cpuidle: using governor menu [ 0.165527] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.166563] PCI: Using configuration type 1 for base access [ 0.167178] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.175309] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.176000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.179108] cryptd: max_cpu_qlen set to 1000 [ 0.181000] ACPI: Added _OSI(Module Device) [ 0.181000] ACPI: Added _OSI(Processor Device) [ 0.181000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.184014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.189764] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197583] ACPI: Interpreter enabled [ 0.198045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.199017] ACPI: Using IOAPIC for interrupt routing [ 0.200131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.201523] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.213146] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216129] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.219050] acpiphp: Slot [2] registered [ 0.220137] acpiphp: Slot [5] registered [ 0.221162] acpiphp: Slot [6] registered [ 0.222163] acpiphp: Slot [3] registered [ 0.223158] acpiphp: Slot [4] registered [ 0.224104] acpiphp: Slot [7] registered [ 0.225117] acpiphp: Slot [8] registered [ 0.226168] acpiphp: Slot [9] registered [ 0.227294] acpiphp: Slot [10] registered [ 0.228102] acpiphp: Slot [11] registered [ 0.229118] acpiphp: Slot [12] registered [ 0.230060] acpiphp: Slot [13] registered [ 0.231081] acpiphp: Slot [14] registered [ 0.232122] acpiphp: Slot [15] registered [ 0.233100] acpiphp: Slot [16] registered [ 0.234101] acpiphp: Slot [17] registered [ 0.235122] acpiphp: Slot [18] registered [ 0.236083] acpiphp: Slot [19] registered [ 0.237154] acpiphp: Slot [20] registered [ 0.238141] acpiphp: Slot [21] registered [ 0.239191] acpiphp: Slot [22] registered [ 0.240073] acpiphp: Slot [23] registered [ 0.241120] acpiphp: Slot [24] registered [ 0.242113] acpiphp: Slot [25] registered [ 0.243098] acpiphp: Slot [26] registered [ 0.244203] acpiphp: Slot [27] registered [ 0.245099] acpiphp: Slot [28] registered [ 0.246142] acpiphp: Slot [29] registered [ 0.247095] acpiphp: Slot [30] registered [ 0.248296] acpiphp: Slot [31] registered [ 0.249065] PCI host bridge to bus 0000:00 [ 0.250028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.251073] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.252025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.253058] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.255019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.257099] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.259257] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.263070] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.266476] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.274014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.278048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.281018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.284018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.286013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.288663] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.290721] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.293043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.297000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.301026] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.311000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.315022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.321243] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.327018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.334026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.348022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.355743] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.363015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.367014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.380015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.390354] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.393325] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.395370] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.396284] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.398173] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.403012] iommu: Default domain type: Passthrough [ 0.404499] SCSI subsystem initialized [ 0.406090] ACPI: bus type USB registered [ 0.407106] usbcore: registered new interface driver usbfs [ 0.409072] usbcore: registered new interface driver hub [ 0.410055] usbcore: registered new device driver usb [ 0.412227] pps_core: LinuxPPS API ver. 1 registered [ 0.414007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.416045] PTP clock support registered [ 0.418408] EDAC MC: Ver: 3.0.0 [ 0.419696] PCI: Using ACPI for IRQ routing [ 0.420451] NetLabel: Initializing [ 0.421005] NetLabel: domain hash size = 128 [ 0.423009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.424106] NetLabel: unlabeled traffic allowed by default [ 0.426244] vgaarb: loaded [ 0.427632] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.429007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.434176] clocksource: Switched to clocksource kvm-clock [ 0.548782] VFS: Disk quotas dquot_6.6.0 [ 0.550707] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.554978] *** VALIDATE ramfs *** [ 0.556236] *** VALIDATE hugetlbfs *** [ 0.557590] pnp: PnP ACPI init [ 0.559975] pnp: PnP ACPI: found 6 devices [ 0.576510] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.579853] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.582076] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.584196] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.587105] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.590131] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.593320] NET: Registered protocol family 2 [ 0.596439] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.602112] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.605872] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.611434] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.615472] TCP: Hash tables configured (established 65536 bind 65536) [ 0.618617] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.621959] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.625163] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.628480] NET: Registered protocol family 1 [ 0.631272] RPC: Registered named UNIX socket transport module. [ 0.633924] RPC: Registered udp transport module. [ 0.635858] RPC: Registered tcp transport module. [ 0.637770] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.640409] NET: Registered protocol family 44 [ 0.641916] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.644420] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.647089] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.649657] PCI: CLS 0 bytes, default 64 [ 0.651408] Unpacking initramfs... [ 2.276539] debug: unmapping init [mem 0xffff9ddbfcc64000-0xffff9ddbfffcffff] [ 2.295305] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.297600] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.300533] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.331796] Initialise system trusted keyrings [ 3.333558] Key type blacklist registered [ 3.335532] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.345172] zbud: loaded [ 3.348331] *** VALIDATE nfs *** [ 3.349636] *** VALIDATE nfs4 *** [ 3.351368] pstore: using deflate compression [ 3.355420] Platform Keyring initialized [ 3.481649] NET: Registered protocol family 38 [ 3.483849] Key type asymmetric registered [ 3.485614] Asymmetric key parser 'x509' registered [ 3.487712] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.491078] io scheduler mq-deadline registered [ 3.496494] io scheduler kyber registered [ 3.498244] io scheduler bfq registered [ 3.500626] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.504229] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.508246] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.511701] ACPI: Power Button [PWRF] [ 3.520933] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.535889] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.597254] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.630310] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.670080] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.675416] Non-volatile memory driver v1.3 [ 3.677177] Linux agpgart interface v0.103 [ 3.711640] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.714675] vda: detected capacity change from 0 to 75087872 [ 3.742683] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.746448] vdb: detected capacity change from 0 to 1073741824 [ 3.834862] libphy: Fixed MDIO Bus: probed [ 3.847857] usbcore: registered new interface driver usbserial_generic [ 3.850732] usbserial: USB Serial support registered for generic [ 3.854145] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.860416] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.862618] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.865772] mousedev: PS/2 mouse device common for all mice [ 3.869601] rtc_cmos 00:05: RTC can wake from S4 [ 3.874593] rtc_cmos 00:05: registered as rtc0 [ 3.876382] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.880652] intel_pstate: CPU model not supported [ 3.884027] hid: raw HID events driver (C) Jiri Kosina [ 3.884324] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.889471] usbcore: registered new interface driver usbhid [ 3.896949] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.898841] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.903230] usbhid: USB HID core driver [ 3.909315] drop_monitor: Initializing network drop monitor service [ 3.911365] Initializing XFRM netlink socket [ 3.913070] NET: Registered protocol family 10 [ 3.916051] Segment Routing with IPv6 [ 3.917324] NET: Registered protocol family 17 [ 3.919144] mpls_gso: MPLS GSO support [ 3.924176] RAS: Correctable Errors collector initialized. [ 3.926312] AVX version of gcm_enc/dec engaged. [ 3.927655] AES CTR mode by8 optimization enabled [ 4.007826] sched_clock: Marking stable (4007755368, 0)->(4974593397, -966838029) [ 4.011527] registered taskstats version 1 [ 4.012940] Loading compiled-in X.509 certificates [ 4.015087] zswap: loaded using pool lzo/zbud [ 4.045926] Key type big_key registered [ 4.062851] Key type encrypted registered [ 4.064509] ima: No TPM chip found, activating TPM-bypass! [ 4.067429] ima: Allocated hash algorithm: sha1 [ 4.069255] ima: No architecture policies found [ 4.070962] evm: Initialising EVM extended attributes: [ 4.072662] evm: security.selinux [ 4.073553] evm: security.ima [ 4.074774] evm: security.capability [ 4.076213] evm: HMAC attrs: 0x1 [ 4.078633] rtc_cmos 00:05: setting system clock to 2026-09-08 01:17:28 UTC (1788830248) [ 4.085775] debug: unmapping init [mem 0xffffffffba803000-0xffffffffba9fffff] [ 4.088914] debug: unmapping init [mem 0xffffffffb9582000-0xffffffffb9858fff] [ 4.094163] Write protecting the kernel read-only data: 28672k [ 4.097887] debug: unmapping init [mem 0xffffffffb7c03000-0xffffffffb7dfffff] [ 4.102122] debug: unmapping init [mem 0xffffffffb8514000-0xffffffffb85fffff] [ 4.150825] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.167806] systemd[1]: Detected virtualization kvm. [ 4.173251] systemd[1]: Detected architecture x86-64. [ 4.178451] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.212697] systemd[1]: No hostname configured. [ 4.218691] systemd[1]: Set hostname to . [ 4.282714] random: systemd: uninitialized urandom read (16 bytes read) [ 4.289441] systemd[1]: Initializing machine ID from random generator. [ 4.678689] random: systemd: uninitialized urandom read (16 bytes read) [ 4.681826] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.687204] random: systemd: uninitialized urandom read (16 bytes read) [ 4.689899] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.695165] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.513484] device-mapper: uevent: version 1.0.3 [ 6.516229] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ 8.484868] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ 8.579990] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 8.895166] scsi host0: ata_piix [ 9.003163] scsi host1: ata_piix [ 9.004591] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 9.007267] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.708106] random: crng init done [ 15.713328] random: 7 urandom warning(s) missed due to ratelimiting [ 17.991537] dracut-initqueue[566]: RTNETLINK answers: File exists [ 18.865669] hrtimer: interrupt took 17648205 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.255596] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.869371] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.369129] SELinux: Disabled at runtime. [ 23.608949] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.619324] systemd[1]: Detected virtualization kvm. [ 23.621523] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.499534] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.556860] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.566239] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.570129] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.574367] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.585153] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.604881] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ 24.866441] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 26.734657] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.565203] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.688683] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.776451] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.449271] EDAC sbridge: Ver: 1.1.2 [ 32.229402] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 32.666762] NFS: Registering the id_resolver key type [ 32.669780] Key type id_resolver registered [ 32.672239] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg226-client login: [ 109.072464] libcfs: loading out-of-tree module taints kernel. [ 109.363095] Key type ._llcrypt registered [ 109.368326] Key type .llcrypt registered [ 109.733104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 109.741121] alg: No test for adler32 (adler32-zlib) [ 111.004552] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 111.686682] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 113.424160] Key type lgssc registered [ 115.010570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 224.895142] Lustre: Mounted lustre-client - version 2.17.57_104_ga33798f [ 229.654031] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.941263] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing check_logdir /tmp/testlogs/ [ 250.054894] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing yml_node [ 250.336995] Lustre: lustre-OST0000-osc-ffff9ddc47009800: disconnect after 23s idle [ 253.268527] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 255.217564] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 257.318298] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 258.512616] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 21:21:41 EDT 2026 [ 273.025633] Lustre: Unmounted lustre-client [ 290.596637] Key type lgssc unregistered [ 290.845669] LNet: 8409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.850646] LNetError: 8409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.867907] LNet: Removed LNI 192.168.202.26@tcp [ 291.630182] Key type .llcrypt unregistered [ 291.636217] Key type ._llcrypt unregistered [ 314.363330] Key type ._llcrypt registered [ 314.369565] Key type .llcrypt registered [ 314.520975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 314.537523] alg: No test for adler32 (adler32-zlib) [ 315.453371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 315.560051] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 315.587965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 319.483917] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 319.686099] LNet: 9282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.697985] LNetError: 9282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 319.724657] LNet: Removed LNI 192.168.202.26@tcp [ 320.613869] Key type .llcrypt unregistered [ 320.617427] Key type ._llcrypt unregistered [ 326.441988] Key type ._llcrypt registered [ 326.444576] Key type .llcrypt registered [ 327.849953] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:22:50 (1788830570) [ 328.526200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 328.553593] alg: No test for adler32 (adler32-zlib) [ 329.477466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.555177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.574282] LNet: 9696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.603373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.656757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 329.672487] LNet: 9703:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 336.502467] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:22:59 (1788830579) [ 336.560457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.585507] LNet: 10275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.615353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.727372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 342.926757] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:23:06 (1788830586) [ 343.302657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.329608] LNet: 10867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.347465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.464982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 350.271771] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:23:13 (1788830593) [ 350.436693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.473527] LNet: 11451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.486022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.524447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 355.875674] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:23:19 (1788830599) [ 356.236912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.276326] LNet: 12035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.299504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.411373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 356.990481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 363.194877] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:23:26 (1788830606) [ 363.395616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.432117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.486946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.810362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 364.131269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 369.323155] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:23:32 (1788830612) [ 369.516970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.548325] LNet: 13222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.553781] LNet: 13222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 369.595300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.675518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 375.824638] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:23:39 (1788830619) [ 375.936889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.991529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.065383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 376.111155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 376.373914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 376.398876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 376.683564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 376.706820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 376.996844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 377.031231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 377.417727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 377.448196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 377.710571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 377.746980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 378.121954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 378.187447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 378.566087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 378.606721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 378.957885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 378.985780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 379.275924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 379.300418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 379.566440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 379.590620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 379.919156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 379.958339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 380.215806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 380.245771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 385.819383] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:23:49 (1788830629) [ 385.967241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.023191] LNet: 14488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.029089] LNet: 14488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 386.052960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.112924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 386.139152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 391.901888] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:23:55 (1788830635) [ 392.056701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.148695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.243680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 392.268865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 398.026551] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:24:01 (1788830641) [ 398.093262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.125695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.232485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 398.262363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 404.589306] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:24:07 (1788830647) [ 404.665069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.722277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.825824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 404.877322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 410.786785] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:24:13 (1788830653) [ 410.998845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.027652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.084506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 411.112521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 417.122924] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:24:20 (1788830660) [ 417.427158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.497378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.606297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 417.648186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 422.966657] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:24:26 (1788830666) [ 423.133220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.171922] LNet: 18007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.177818] LNet: 18007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 423.203690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.297100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 423.652316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 428.660107] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:24:32 (1788830672) [ 428.886720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.952131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.201267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 429.470256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 434.894203] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:24:37 (1788830677) [ 435.008222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.052548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.232454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.426537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.612131] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:24:44 (1788830684) [ 441.873391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.917644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.376417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.675797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.973730] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:24:51 (1788830691) [ 448.051684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.087052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.158815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.194881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 455.022939] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:24:58 (1788830698) [ 455.138949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.193984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.307111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.360945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.910592] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:25:04 (1788830704) [ 461.185477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.233469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.313475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 466.733308] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:25:09 (1788830709) [ 467.021775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.081338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.164533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 473.454464] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:25:16 (1788830716) [ 473.662909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.720574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.799534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 474.421788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 480.562891] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:25:23 (1788830723) [ 480.724502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.802339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.901723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 480.948654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 486.431495] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:25:29 (1788830729) [ 486.633914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.665822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.726443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 486.776630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 492.452917] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:25:35 (1788830735) [ 492.594778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.616470] LNet: 24612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.622273] LNet: 24612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 492.656188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.751935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 492.786458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 499.709850] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:25:42 (1788830742) [ 499.810165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.869374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.985247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 500.028540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 505.334049] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:25:48 (1788830748) [ 505.487335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.537247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.602371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 505.658337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 505.675559] LNetError: 25802:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 505.698390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 505.721576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 510.882646] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:25:53 (1788830753) [ 510.976398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.049785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.177809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.aB2g6SL48J [ 511.243924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.aB2g6SL48J [ 511.288622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.aB2g6SL48J [ 517.095402] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:26:00 (1788830760) [ 517.182297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.226682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.301687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 517.332295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 517.398606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 523.062438] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 21:26:06 (1788830766) [ 523.166548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.219673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.289730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 523.313303] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 524.662332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 525.397708] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 527.152170] Key type lgssc registered [ 528.115868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 530.613018] LNetError: 27563:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.26@tcp99 has no available nets [ 532.487375] Key type lgssc unregistered [ 532.774276] LNetError: 28668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 533.800730] LNet: Removed LNI 192.168.202.26@tcp [ 534.241727] Key type .llcrypt unregistered [ 534.245259] Key type ._llcrypt unregistered [ 539.328885] Key type ._llcrypt registered [ 539.330568] Key type .llcrypt registered [ 540.386528] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:26:23 (1788830783) [ 540.887962] Key type .llcrypt unregistered [ 540.890471] Key type ._llcrypt unregistered [ 545.946689] Key type ._llcrypt registered [ 545.954385] Key type .llcrypt registered [ 546.116064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 546.131473] alg: No test for adler32 (adler32-zlib) [ 547.040568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.118597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 547.141807] LNet: 29578:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 547.150725] LNet: 29578:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 547.159559] LNet: 29578:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 547.185852] LNet: Message forwarding will be enabled in 2s [ 550.304452] LNet: Message forwarding enabled [ 550.617370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 550.630490] LNet: Message forwarding will stop in 2s to 6s [ 554.405550] LNet: Message forwarding disabled [ 555.070879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 555.089469] LNet: 29676:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 555.101856] LNet: 29676:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 555.108253] LNet: 29676:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 555.118639] LNet: Message forwarding will be enabled in 2s [ 555.269586] LNet: 29689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.762661] Key type .llcrypt unregistered [ 555.764700] Key type ._llcrypt unregistered [ 565.678233] Key type ._llcrypt registered [ 565.681328] Key type .llcrypt registered [ 567.124575] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:26:49 (1788830809) [ 567.528241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 567.544661] alg: No test for adler32 (adler32-zlib) [ 568.443207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.511403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 574.615948] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:26:57 (1788830817) [ 574.743423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.776787] LNet: 31087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.804736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.878062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 574.942848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 575.004228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 575.045801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 575.089239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 575.134691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 575.169843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 575.199897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 575.227270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 575.257181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 575.284403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 575.316170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 575.347242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 575.369823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 575.396433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 575.414705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 575.446426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 575.480411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 575.516136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 575.530259] LNetError: 31133:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 583.719151] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:27:05 (1788830825) [ 583.988390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.103488] LNet: 31701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.179216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.945282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 597.532264] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:27:20 (1788830840) [ 600.518683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg226-server [ 603.170766] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:27:25 (1788830845) [ 603.658227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.720486] LNet: 32638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 603.764512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.975511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 604.057663] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 606.068382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[27-31]@tcp [ 606.106678] LNet: 32771:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 606.173638] LNetError: 32649:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.29@tcp has gone from up to down [ 606.254113] LNetError: 32642:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.27@tcp is being used as a gateway but routing feature is not turned on [ 618.045574] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:27:39 (1788830859) [ 622.279693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg226-server [ 625.428347] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:27:47 (1788830867) [ 625.750026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.806753] LNet: 33709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.828436] LNetError: 33709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.877346] LNet: Removed LNI 192.168.202.26@tcp [ 625.969462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.066047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 626.101663] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 627.220531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[27-32/2]@tcp [ 627.294950] LNet: 33826:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 627.319095] LNet: 33826:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 627.353926] LNetError: 33718:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.29@tcp has gone from up to down [ 627.387764] LNetError: 33718:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 627.394571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[27-32/2]@tcp [ 627.420444] LNetError: 33713:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.27@tcp is being used as a gateway but routing feature is not turned on [ 627.420457] LNetError: 33713:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 636.991873] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:27:59 (1788830879) [ 637.632860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.672808] LNet: 34419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.683401] LNetError: 34419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.706958] LNet: Removed LNI 192.168.202.26@tcp [ 637.749759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.954962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 638.193951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.339776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.586988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.739358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.852562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.001283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 639.248397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.322749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.541095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.719129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.803617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.920926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 640.112757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.235238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.443868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 640.641236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.751330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.887168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 641.126167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.280347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.405559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 641.502742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.555308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.595500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 641.660372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.697088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.757551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 641.845822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.885744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.958904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 642.046766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.134703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.236817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 651.493911] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:28:14 (1788830894) [ 654.232502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg226-server [ 657.077992] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:28:19 (1788830899) [ 659.604670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg226-server [ 662.214555] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:28:24 (1788830904) [ 662.370633] LNet: 35815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.374969] LNet: 35815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 663.251554] Key type .llcrypt unregistered [ 663.263609] Key type ._llcrypt unregistered [ 670.053139] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 670.298914] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 670.301780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 670.311209] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 672.933408] Key type ._llcrypt registered [ 672.936162] Key type .llcrypt registered [ 673.137798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 673.147784] alg: No test for adler32 (adler32-zlib) [ 674.156586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.252901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.314757] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 674.991976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 675.020551] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 677.083650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 677.101259] LNet: Removed LNI 10.1.2.3@tcp [ 677.251938] LNet: 36537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.260467] LNetError: 36537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.296406] LNet: Removed LNI 192.168.202.26@tcp [ 678.099971] Key type .llcrypt unregistered [ 678.101771] Key type ._llcrypt unregistered [ 691.499441] Key type ._llcrypt registered [ 691.501439] Key type .llcrypt registered [ 692.728890] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:28:55 (1788830935) [ 693.481445] Key type .llcrypt unregistered [ 693.483400] Key type ._llcrypt unregistered [ 700.756203] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 701.034975] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 701.040958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 701.056727] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 703.633733] Key type ._llcrypt registered [ 703.637409] Key type .llcrypt registered [ 703.825948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.847706] alg: No test for adler32 (adler32-zlib) [ 704.788978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.861770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.908991] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 705.509612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 705.521875] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 707.570053] LNet: 38025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.579931] LNetError: 38025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 707.585501] LNet: Removed LNI 10.1.2.3@tcp [ 707.599669] LNet: Removed LNI 192.168.202.26@tcp [ 708.224824] Key type .llcrypt unregistered [ 708.227935] Key type ._llcrypt unregistered [ 719.452178] Key type ._llcrypt registered [ 719.454068] Key type .llcrypt registered [ 720.835575] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:29:23 (1788830963) [ 721.315998] Key type .llcrypt unregistered [ 721.323565] Key type ._llcrypt unregistered [ 726.651470] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 726.885174] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 726.887885] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 726.890926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 729.523100] Key type ._llcrypt registered [ 729.524922] Key type .llcrypt registered [ 729.765904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 729.793963] alg: No test for adler32 (adler32-zlib) [ 730.743930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.819309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 730.839159] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 732.565554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 732.584070] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 734.661625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 734.682500] LNet: Removed LNI 10.1.2.3@tcp [ 734.694484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 734.728221] LNet: Removed LNI 10.1.2.31@tcp [ 734.741923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 734.772640] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 735.374517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 735.392890] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 736.415415] LNet: 39942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.428937] LNetError: 39942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 736.438810] LNet: Removed LNI 10.1.2.31@tcp [ 736.456533] LNet: Removed LNI 10.1.2.3@tcp [ 737.106363] Key type .llcrypt unregistered [ 737.108107] Key type ._llcrypt unregistered [ 748.858422] Key type ._llcrypt registered [ 748.861202] Key type .llcrypt registered [ 750.214567] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:29:53 (1788830993) [ 750.783745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 750.800743] alg: No test for adler32 (adler32-zlib) [ 751.721965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.807603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 751.841728] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 752.523767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.541264] LNet: 40892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.547052] LNetError: 40892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.560667] LNet: Removed LNI 192.168.202.26@tcp [ 752.578380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.621650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.26@tcp [ 752.642340] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 753.241650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.26@tcp [ 753.265995] LNet: Removed LNI 192.168.202.26@tcp [ 753.329757] LNet: 41027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.950665] Key type .llcrypt unregistered [ 753.953482] Key type ._llcrypt unregistered [ 766.901924] Key type ._llcrypt registered [ 766.904756] Key type .llcrypt registered [ 768.134603] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:30:11 (1788831011) [ 770.905967] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg226-server [ 772.923870] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:30:15 (1788831015) [ 773.591324] Key type .llcrypt unregistered [ 773.593813] Key type ._llcrypt unregistered [ 781.024041] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 781.115115] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 781.135345] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 781.150584] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 783.879719] Key type ._llcrypt registered [ 783.883442] Key type .llcrypt registered [ 784.135357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.157428] alg: No test for adler32 (adler32-zlib) [ 785.095402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.154525] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 785.861632] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 788.146159] LNet: 42858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.151787] LNetError: 42858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.155926] LNet: Removed LNI 192.168.202.26@tcp1 [ 788.164421] LNet: Removed LNI 10.1.2.3@tcp2 [ 788.782635] Key type .llcrypt unregistered [ 788.784294] Key type ._llcrypt unregistered [ 802.594918] Key type ._llcrypt registered [ 802.599287] Key type .llcrypt registered [ 803.927701] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:30:46 (1788831046) [ 804.564909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.575645] alg: No test for adler32 (adler32-zlib) [ 805.540808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.613079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.647419] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 806.385448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.413817] LNet: 43826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.421241] LNetError: 43826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.436621] LNet: Removed LNI 192.168.202.26@tcp [ 806.466868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.553643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 806.611273] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 813.828786] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:30:56 (1788831056) [ 814.169497] LNet: 44417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.184385] LNetError: 44417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.201180] LNet: Removed LNI 192.168.202.26@tcp [ 814.845837] Key type .llcrypt unregistered [ 814.849943] Key type ._llcrypt unregistered [ 821.319920] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.525552] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 821.549397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 821.561459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 824.241841] Key type ._llcrypt registered [ 824.243750] Key type .llcrypt registered [ 824.498615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 824.537707] alg: No test for adler32 (adler32-zlib) [ 825.529578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.574846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 825.596767] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 826.383735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 826.401754] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 828.943549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 828.968440] LNet: 45150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.975735] LNetError: 45150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 828.985095] LNet: Removed LNI 10.1.2.3@tcp [ 828.999125] LNet: Removed LNI 192.168.202.26@tcp [ 829.030851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.101568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 829.128329] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 829.134319] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 829.836412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.863534] LNet: 45177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.872349] LNetError: 45177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 829.880415] LNetError: 45177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 829.888389] LNet: Removed LNI 10.1.2.3@tcp [ 829.899387] LNet: Removed LNI 192.168.202.26@tcp [ 829.919469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.003525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 830.051061] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 830.057213] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 830.881292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 830.911806] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 831.998730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.014751] LNet: 45340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.019076] LNetError: 45340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 832.024572] LNetError: 45340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 832.029558] LNet: Removed LNI 10.1.2.3@tcp [ 832.031940] LNet: Removed LNI 192.168.202.26@tcp [ 832.044583] LNet: Removed LNI 192.168.202.26@tcp2 [ 832.069312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.146538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 832.166894] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 832.171899] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 832.177887] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 840.735446] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:31:23 (1788831083) [ 840.835202] LNet: 45928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.853076] LNetError: 45928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 840.862316] LNetError: 45928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 840.877135] LNet: Removed LNI 10.1.2.3@tcp [ 840.879264] LNet: Removed LNI 192.168.202.26@tcp [ 840.893820] LNet: Removed LNI 192.168.202.26@tcp2 [ 841.702671] Key type .llcrypt unregistered [ 841.707112] Key type ._llcrypt unregistered [ 847.976917] Key type ._llcrypt registered [ 847.982423] Key type .llcrypt registered [ 848.292622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 848.307430] alg: No test for adler32 (adler32-zlib) [ 849.246093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.294458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 849.313691] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 850.266876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.288780] LNet: 46328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.294381] LNetError: 46328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.308854] LNet: Removed LNI 192.168.202.26@tcp [ 850.341150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.424153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 850.482863] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 850.949942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.970495] LNet: 46358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.975672] LNetError: 46358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.995375] LNet: Removed LNI 192.168.202.26@tcp [ 851.019507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.142968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 851.191623] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 851.718985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.746592] LNet: Removed LNI 192.168.202.26@tcp [ 851.764111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.814175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 851.837115] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 852.374312] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 852.600342] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 852.612640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 852.662734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 854.990562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.014485] LNet: 46467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.024906] LNet: 46467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 855.049431] LNetError: 46467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.062553] LNetError: 46467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 855.092891] LNet: Removed LNI 192.168.202.26@tcp [ 855.126133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.236131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 855.309128] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 863.447315] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:31:46 (1788831106) [ 863.934739] LNet: 47064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.941077] LNetError: 47064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.977573] LNet: Removed LNI 192.168.202.26@tcp [ 864.877395] Key type .llcrypt unregistered [ 864.880431] Key type ._llcrypt unregistered [ 873.075853] Key type ._llcrypt registered [ 873.084217] Key type .llcrypt registered [ 873.303208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 873.329155] alg: No test for adler32 (adler32-zlib) [ 874.275516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.367692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 874.404947] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 875.612953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.667977] LNet: 47460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.680541] LNetError: 47460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.716517] LNet: Removed LNI 192.168.202.26@tcp [ 875.806178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.877946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 875.895629] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 882.781916] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:32:05 (1788831125) [ 883.049263] LNet: 48052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.055771] LNetError: 48052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.078342] LNet: Removed LNI 192.168.202.26@tcp [ 883.878486] Key type .llcrypt unregistered [ 883.888377] Key type ._llcrypt unregistered [ 890.718586] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 890.962432] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 890.970317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 890.973933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 893.624946] Key type ._llcrypt registered [ 893.638239] Key type .llcrypt registered [ 894.022094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.040990] alg: No test for adler32 (adler32-zlib) [ 895.161684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.304645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.364912] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 896.774403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.816380] LNet: 48493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.823941] LNetError: 48493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.853565] LNet: Removed LNI 192.168.202.26@tcp [ 896.904858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.997358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 897.017880] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 905.711452] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:32:28 (1788831148) [ 905.947498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.971686] LNet: 49093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.979507] LNetError: 49093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.993523] LNet: Removed LNI 192.168.202.26@tcp [ 906.013606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.107783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 913.836474] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:32:36 (1788831156) [ 914.113916] LNet: 49674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.893795] Key type .llcrypt unregistered [ 914.900414] Key type ._llcrypt unregistered [ 922.839329] Key type ._llcrypt registered [ 922.845112] Key type .llcrypt registered [ 923.030081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.050314] alg: No test for adler32 (adler32-zlib) [ 924.027499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.088398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 924.112339] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 925.176623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.201632] LNet: 50070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.207317] LNetError: 50070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.232209] LNet: Removed LNI 192.168.202.26@tcp [ 925.272428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.414414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 925.449280] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 933.278700] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:32:56 (1788831176) [ 933.492997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.533229] LNet: 50663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.539099] LNetError: 50663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.571563] LNet: Removed LNI 192.168.202.26@tcp [ 933.607890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.730927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 941.820335] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:33:04 (1788831184) [ 943.691900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 945.954973] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:33:08 (1788831188) [ 947.844991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 949.726568] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:33:12 (1788831192) [ 951.702230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 953.451615] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:33:16 (1788831196) [ 955.491462] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 957.360948] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:33:20 (1788831200) [ 959.635425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 962.169300] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:33:24 (1788831204) [ 962.321380] LNet: 52711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.315568] Key type .llcrypt unregistered [ 963.317293] Key type ._llcrypt unregistered [ 971.794290] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 972.034313] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 972.048893] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 972.061578] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 974.994649] Key type ._llcrypt registered [ 975.009814] Key type .llcrypt registered [ 975.282850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.304525] alg: No test for adler32 (adler32-zlib) [ 976.360715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.528452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 976.576602] LNet: Added LNI 192.168.202.26@tcp [8/512/0/180] [ 976.585629] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 976.590233] LNet: 53036:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 976.722365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.789494] LNet: 53055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.807438] LNetError: 53055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 976.823834] LNet: Removed LNI 10.1.2.3@tcp [ 976.828044] LNetError: 53055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.860303] LNet: Removed LNI 192.168.202.26@tcp [ 976.918937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.046182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 977.079535] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 977.179945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.212540] LNet: 53080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.218887] LNetError: 53080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.231593] LNet: Removed LNI 192.168.202.26@tcp [ 977.295678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.390381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 977.434471] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 977.448740] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 977.451457] LNet: 53088:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 977.618597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.652299] LNet: 53106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.670500] LNetError: 53106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 977.683080] LNet: Removed LNI 10.1.2.3@tcp [ 977.685221] LNetError: 53106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.725571] LNet: Removed LNI 192.168.202.26@tcp [ 977.787477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.884720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 977.919700] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 977.924902] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 977.931800] LNet: 53114:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 978.190730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.209671] LNet: 53140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.215419] LNetError: 53140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 978.220660] LNet: Removed LNI 10.1.2.3@tcp [ 978.222973] LNetError: 53140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.237967] LNet: Removed LNI 192.168.202.26@tcp [ 978.254206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.304722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.322725] LNet: Added LNI 192.168.202.26@tcp [8/256/0/182] [ 978.327056] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 978.396332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.416836] LNet: 53168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.421723] LNetError: 53168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 978.426678] LNet: Removed LNI 10.1.2.3@tcp [ 978.430582] LNetError: 53168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.449734] LNet: Removed LNI 192.168.202.26@tcp [ 978.482275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.550777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.580114] LNet: Added LNI 192.168.202.26@tcp [8/256/0/182] [ 978.584584] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 978.587590] LNet: 53176:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 978.684660] LNet: Removed LNI 10.1.2.3@tcp [ 978.701166] LNet: Removed LNI 192.168.202.26@tcp [ 979.393823] Key type .llcrypt unregistered [ 979.396593] Key type ._llcrypt unregistered [ 993.314401] Key type ._llcrypt registered [ 993.317924] Key type .llcrypt registered [ 995.085213] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 21:33:57 (1788831237) [ 997.949470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1000.385512] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:34:02 (1788831242) [ 1000.903309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1000.931821] alg: No test for adler32 (adler32-zlib) [ 1001.818406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.921775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1001.981411] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1003.231897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.26 [ 1003.270918] LNet: 54439:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1003.289801] LNet: Removed LNI 192.168.202.26@tcp [ 1012.038672] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:34:14 (1788831254) [ 1012.583950] LNet: 55010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.590316] LNetError: 55010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.628850] LNet: Removed LNI 192.168.202.26@tcp [ 1013.730602] Key type .llcrypt unregistered [ 1013.732060] Key type ._llcrypt unregistered [ 1023.962153] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1024.313871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1027.452613] Key type ._llcrypt registered [ 1027.455831] Key type .llcrypt registered [ 1027.700928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.715862] alg: No test for adler32 (adler32-zlib) [ 1029.238740] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.234254] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:34:39 (1788831279) [ 1037.381929] LNet: 55914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.413385] LNetError: 55914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.450023] LNet: Removed LNI 10.1.2.3@tcp [ 1038.526422] Key type .llcrypt unregistered [ 1038.530877] Key type ._llcrypt unregistered [ 1045.615239] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1045.871913] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1045.883800] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1045.889887] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1048.799380] Key type ._llcrypt registered [ 1048.804406] Key type .llcrypt registered [ 1048.975540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1048.995986] alg: No test for adler32 (adler32-zlib) [ 1049.985108] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1056.847841] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:34:59 (1788831299) [ 1058.887723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1060.983844] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:35:03 (1788831303) [ 1061.245771] LNet: 57108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.251486] LNetError: 57108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1061.278597] LNet: Removed LNI 10.1.2.3@tcp [ 1062.216394] Key type .llcrypt unregistered [ 1062.218582] Key type ._llcrypt unregistered [ 1070.150651] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1070.445486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1070.449747] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1070.456291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1073.382443] Key type ._llcrypt registered [ 1073.385176] Key type .llcrypt registered [ 1073.576794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1073.595603] alg: No test for adler32 (adler32-zlib) [ 1074.590414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.681499] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1084.604325] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:35:27 (1788831327) [ 1085.324249] LNet: 58338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.342424] LNetError: 58338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1085.363072] LNet: Removed LNI 10.1.2.3@tcp [ 1086.075491] Key type .llcrypt unregistered [ 1086.077966] Key type ._llcrypt unregistered [ 1093.489611] Key type ._llcrypt registered [ 1093.491188] Key type .llcrypt registered [ 1093.757606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1093.797170] alg: No test for adler32 (adler32-zlib) [ 1094.851654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1094.885655] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1102.871438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 1103.687554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_interrupt -r 1 [ 1103.729375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_interrupt -r 1 [ 1103.768467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1103.786408] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1104.075571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.125709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.160353] LNet: 1 local NIs in recovery (showing 1): 192.168.202.26@tcp [ 1104.171894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.203981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.234666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.264816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.291820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.331138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.357610] LNet: 58773:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.375795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.402810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.445297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.488338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.544135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.572693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.618272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.410627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_dropped -r 1 [ 1105.477542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_dropped -r 1 [ 1105.534643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1105.571166] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1105.579792] LNet: Skipped 1 previous similar message [ 1105.937942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.978263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.003432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.025488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.056144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.083801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.111150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.149098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.165955] LNet: 58883:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1106.190424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.223427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.274575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.346199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.409560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.444689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.476259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1106.932326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_aborted -r 1 [ 1106.965705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_aborted -r 1 [ 1106.999376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1107.013849] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1107.220574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.251690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.274382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.301230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.334857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.401726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.457788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.487517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.500610] LNet: 58992:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.516770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.549797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.597224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.643304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.697809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.778330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.848638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.581547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_no_route -r 1 [ 1108.620070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_no_route -r 1 [ 1108.660931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1108.951659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.998338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.035492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.085131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.140883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.189821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.226383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.268660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.311429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.381859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.425241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.482598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.517648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.551385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.597972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1110.298137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_timeout -r 1 [ 1110.337240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_timeout -r 1 [ 1110.376679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1110.414163] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1110.422396] LNet: Skipped 2 previous similar messages [ 1110.710428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.763576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.839272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.886243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.955474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.020997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.075872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.118756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.159560] LNet: 59211:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1111.170521] LNet: 59211:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1111.190079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.235539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.272493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.326452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.368695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.417766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.468086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1112.161782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_error -r 1 [ 1112.214255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_error -r 1 [ 1112.251366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1112.602989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.633705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1112.670246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.715945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.790069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.833112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.874250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.941080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.003285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.057829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.101376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.131406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.165269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.191500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.214428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.506402] LNet: 59361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.519437] LNetError: 59361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.534515] LNet: Removed LNI 192.168.202.26@tcp [ 1114.096215] Key type .llcrypt unregistered [ 1114.099145] Key type ._llcrypt unregistered [ 1133.677819] Key type ._llcrypt registered [ 1133.681187] Key type .llcrypt registered [ 1135.146921] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:36:17 (1788831377) [ 1136.129434] Key type .llcrypt unregistered [ 1136.134657] Key type ._llcrypt unregistered [ 1142.870350] Key type ._llcrypt registered [ 1142.871874] Key type .llcrypt registered [ 1143.129935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1143.164970] alg: No test for adler32 (adler32-zlib) [ 1144.152337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1144.186052] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1153.066752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 1165.035692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.087651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1165.110570] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1166.768703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_interrupt -r 1 [ 1166.811934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_interrupt -r 1 [ 1166.866422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_interrupt -r 1 [ 1166.913866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_interrupt -r 1 [ 1166.977682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_interrupt -r 1 [ 1167.023908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_interrupt -r 1 [ 1167.074290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_interrupt -r 1 [ 1167.117985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_interrupt -r 1 [ 1167.175927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1167.217750] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1167.227811] LNet: 1 local NIs in recovery (showing 1): 192.168.202.26@tcp [ 1167.494757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.530397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.573896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.616042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.651976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.688797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.722225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.754485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.771981] LNet: 60828:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.794194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.836566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.868690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.909938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.951671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.998419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.054741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1168.757131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_dropped -r 1 [ 1168.806596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_dropped -r 1 [ 1168.876574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_dropped -r 1 [ 1168.956739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_dropped -r 1 [ 1169.016949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_dropped -r 1 [ 1169.097900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_dropped -r 1 [ 1169.187342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_dropped -r 1 [ 1169.208426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_dropped -r 1 [ 1169.279525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1169.308026] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1169.323630] LNet: Skipped 2 previous similar messages [ 1169.695864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.740552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.810266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.886835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.922557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.946629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.005656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.054057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.092494] LNet: 60949:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.124805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.193730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.235275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.256223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.281381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.316698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.350280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1170.831846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_aborted -r 1 [ 1170.866818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_aborted -r 1 [ 1170.904692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_aborted -r 1 [ 1170.930483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_aborted -r 1 [ 1170.956843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_aborted -r 1 [ 1170.986361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_aborted -r 1 [ 1171.016221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_aborted -r 1 [ 1171.053827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_aborted -r 1 [ 1171.088771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1171.118747] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1171.122990] LNet: Skipped 2 previous similar messages [ 1171.338333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.365122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.396967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.438724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.466503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.501757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.557085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.602751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.636959] LNet: 61071:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.656117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.702515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.740607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.773973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.829593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.880846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.914071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1172.376389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_no_route -r 1 [ 1172.410063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_no_route -r 1 [ 1172.436156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_no_route -r 1 [ 1172.464605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_no_route -r 1 [ 1172.498951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_no_route -r 1 [ 1172.530672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_no_route -r 1 [ 1172.578373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_no_route -r 1 [ 1172.610709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_no_route -r 1 [ 1172.661992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1172.992657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.022461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.053534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.081234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.110882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.156985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.189355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.227693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.250751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.282883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.311964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.340867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.366974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.396616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.431719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1173.940045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_timeout -r 1 [ 1173.991679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_timeout -r 1 [ 1174.057759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_timeout -r 1 [ 1174.092147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_timeout -r 1 [ 1174.155895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_timeout -r 1 [ 1174.244214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_timeout -r 1 [ 1174.308839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_timeout -r 1 [ 1174.368272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_timeout -r 1 [ 1174.429743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1174.470196] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1174.478398] LNet: Skipped 5 previous similar messages [ 1174.747686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.807062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.870807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.916489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.942072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.997916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.050307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.105730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.135546] LNet: 61313:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.146515] LNet: 61313:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1175.170775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.214641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.259721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.294871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.323627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.390622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.461941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1176.035560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_error -r 1 [ 1176.090505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_error -r 1 [ 1176.121079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_error -r 1 [ 1176.162557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_error -r 1 [ 1176.200529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_error -r 1 [ 1176.243873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_error -r 1 [ 1176.281712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_error -r 1 [ 1176.341922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_error -r 1 [ 1176.381781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1176.792878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.846250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.902282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.938950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.966526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.012691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.044742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.079657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.137812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.168627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.211595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.250506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.292110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.336815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.387803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.640874] LNet: 61511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.652089] LNetError: 61511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.662754] LNet: Removed LNI 192.168.202.26@tcp [ 1183.673391] LNet: Removed LNI 192.168.202.26@tcp1 [ 1184.127855] Key type .llcrypt unregistered [ 1184.134096] Key type ._llcrypt unregistered [ 1206.137496] Key type ._llcrypt registered [ 1206.139801] Key type .llcrypt registered [ 1207.668631] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:37:30 (1788831450) [ 1208.835842] Key type .llcrypt unregistered [ 1208.843530] Key type ._llcrypt unregistered [ 1218.107464] Key type ._llcrypt registered [ 1218.110116] Key type .llcrypt registered [ 1218.387058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1218.416658] alg: No test for adler32 (adler32-zlib) [ 1219.439870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1219.477724] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1228.264685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 1229.080337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e remote_dropped -r 1 [ 1229.122746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e remote_dropped -r 1 [ 1229.164372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1229.204259] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1229.511809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.574747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.643029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.732861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.789521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.861491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.936333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.994798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.047865] LNet: 62759:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.098300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.156284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.198810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.254979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.309490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.380302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.453433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1231.240491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e remote_error -r 1 [ 1231.339184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e remote_error -r 1 [ 1231.404624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1231.427187] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1231.866397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1231.926771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1231.990860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.046151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.103947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.152487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.218182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.269308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.304167] LNet: 62868:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.343704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.402972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.437167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.481822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1232.519600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1232.570448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1232.621248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1233.281526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e remote_timeout -r 1 [ 1233.307582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e remote_timeout -r 1 [ 1233.334447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1233.350230] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1233.571326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1233.623229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1233.680124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1233.720418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1233.771583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1233.818408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1233.861450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1233.888964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1233.916682] LNet: 62978:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1233.953940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.030659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.093995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.155295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.199858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.251700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.303482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1234.655244] LNet: 63019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1234.661496] LNetError: 63019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.686261] LNet: Removed LNI 192.168.202.26@tcp [ 1236.263539] Key type .llcrypt unregistered [ 1236.271816] Key type ._llcrypt unregistered [ 1254.998456] Key type ._llcrypt registered [ 1255.005102] Key type .llcrypt registered [ 1256.840462] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:38:19 (1788831499) [ 1257.814391] Key type .llcrypt unregistered [ 1257.817203] Key type ._llcrypt unregistered [ 1264.645686] Key type ._llcrypt registered [ 1264.649164] Key type .llcrypt registered [ 1264.819136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1264.829788] alg: No test for adler32 (adler32-zlib) [ 1265.760042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1265.792674] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1275.726649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 1287.780308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.847111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1287.885719] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1289.533865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e remote_dropped -r 1 [ 1289.581419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e remote_dropped -r 1 [ 1289.615338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e remote_dropped -r 1 [ 1289.665755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e remote_dropped -r 1 [ 1289.712702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e remote_dropped -r 1 [ 1289.768765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e remote_dropped -r 1 [ 1289.824408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e remote_dropped -r 1 [ 1289.864716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e remote_dropped -r 1 [ 1289.911783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1289.933956] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1289.956083] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.126@tcp [ 1290.218389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1290.251726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1290.304432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1290.351876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1290.387985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1290.437171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1290.492693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1290.519934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1290.557609] LNet: 64485:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1290.576878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.611737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.648416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.691406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.731983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.767649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1290.799935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1290.843954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.467328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e remote_error -r 1 [ 1291.509602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e remote_error -r 1 [ 1291.555583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e remote_error -r 1 [ 1291.615045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e remote_error -r 1 [ 1291.657046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e remote_error -r 1 [ 1291.707943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e remote_error -r 1 [ 1291.745865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e remote_error -r 1 [ 1291.784264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e remote_error -r 1 [ 1291.845620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1291.880154] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1291.888400] LNet: Skipped 2 previous similar messages [ 1292.295990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.322354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1292.357966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.391556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.441761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.480503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.517405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.552784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1292.577819] LNet: 64608:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1292.593902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1292.615539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1292.638754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1292.668318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1292.689905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1292.714559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1292.746761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1292.781079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.466705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e remote_timeout -r 1 [ 1293.513514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e remote_timeout -r 1 [ 1293.555744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e remote_timeout -r 1 [ 1293.613586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e remote_timeout -r 1 [ 1293.652520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e remote_timeout -r 1 [ 1293.694115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e remote_timeout -r 1 [ 1293.754353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e remote_timeout -r 1 [ 1293.808658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e remote_timeout -r 1 [ 1293.888729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.126@tcp [ 1293.927189] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1293.934365] LNet: Skipped 1 previous similar message [ 1294.235100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1294.266122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1294.296546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1294.326540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1294.355647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1294.391902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1294.426738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1294.460704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1294.477958] LNet: 64731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1294.502904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1294.536134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1294.572799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1294.596443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1294.632776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1294.674926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1294.720853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1294.785618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1301.129560] LNet: 64810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.136064] LNetError: 64810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.150591] LNet: Removed LNI 192.168.202.26@tcp [ 1301.162670] LNet: Removed LNI 192.168.202.26@tcp1 [ 1301.548700] Key type .llcrypt unregistered [ 1301.551197] Key type ._llcrypt unregistered [ 1323.922195] Key type ._llcrypt registered [ 1323.924844] Key type .llcrypt registered [ 1325.571705] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:39:28 (1788831568) [ 1327.188475] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1328.665844] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:39:31 (1788831571) [ 1329.836450] Key type .llcrypt unregistered [ 1329.838447] Key type ._llcrypt unregistered [ 1337.513396] Key type ._llcrypt registered [ 1337.516490] Key type .llcrypt registered [ 1337.690335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1337.704312] alg: No test for adler32 (adler32-zlib) [ 1338.613122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1338.650177] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1347.243943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 1348.306245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e network_timeout -r 1 [ 1348.368270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e network_timeout -r 1 [ 1348.446313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 1348.484509] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1348.798646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.839659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1348.884907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.897084] LNet: 1 local NIs in recovery (showing 1): 192.168.202.26@tcp [ 1348.912423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1348.946370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1349.017786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1349.094114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1349.150986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1349.171831] LNet: 66351:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1349.195509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1349.235569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1349.314334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.369265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.444121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.537321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.951915] LNet: 66390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.959598] LNetError: 66390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.983633] LNet: Removed LNI 192.168.202.26@tcp [ 1350.347565] Key type .llcrypt unregistered [ 1350.349516] Key type ._llcrypt unregistered [ 1371.916441] Key type ._llcrypt registered [ 1371.922695] Key type .llcrypt registered [ 1372.164175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1372.209846] alg: No test for adler32 (adler32-zlib) [ 1373.255679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1373.303621] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1383.063717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 1393.456717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.519790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1393.576602] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1395.450390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e network_timeout -r 1 [ 1395.509768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e network_timeout -r 1 [ 1395.572991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e network_timeout -r 1 [ 1395.628115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e network_timeout -r 1 [ 1395.675706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e network_timeout -r 1 [ 1395.718322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e network_timeout -r 1 [ 1395.779620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e network_timeout -r 1 [ 1395.839768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e network_timeout -r 1 [ 1395.894736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 1395.912776] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1396.229939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1396.297549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1396.367393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1396.405684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1396.444461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1396.487629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1396.575606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1396.597395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1396.610727] LNet: 67277:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1396.628739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1396.652931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1396.672869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1396.701761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1396.737402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1396.779073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1400.834067] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1400.993061] LNet: 1 local NIs in recovery (showing 1): 192.168.202.26@tcp [ 1401.310792] LNet: 67351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.321076] LNetError: 67351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.335319] LNet: Removed LNI 192.168.202.26@tcp [ 1401.347905] LNet: Removed LNI 192.168.202.26@tcp1 [ 1401.986865] Key type .llcrypt unregistered [ 1401.988527] Key type ._llcrypt unregistered [ 1423.669478] Key type ._llcrypt registered [ 1423.671278] Key type .llcrypt registered [ 1425.254685] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:41:07 (1788831667) [ 1425.956070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1425.995462] alg: No test for adler32 (adler32-zlib) [ 1426.991568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1427.077606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1427.121106] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1428.717939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1428.745313] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1429.895510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 1430.045756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1430.150162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1430.202264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1430.263886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1430.325728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1430.390273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.26@tcp [ 1430.418270] LNetError: 68452:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.26@tcp to recovery [ 1431.137718] LNet: 1 local NIs in recovery (showing 1): 192.168.202.26@tcp [ 1431.150423] LNet: There was an unexpected network error while writing to 192.168.202.26: rc = -22 [ 1433.184552] LNet: There was an unexpected network error while writing to 192.168.202.26: rc = -22 [ 1434.142790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1437.283113] LNet: There was an unexpected network error while writing to 192.168.202.26: rc = -22 [ 1438.047689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1438.083759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1438.177157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1438.192671] LNet: 68560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.197577] LNetError: 68560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.200267] LNet: Removed LNI 192.168.202.26@tcp [ 1439.202929] LNetError: 68560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.227820] LNet: Removed LNI 192.168.202.26@tcp1 [ 1439.299982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.395284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1439.435750] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1440.396887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1440.437734] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1441.472756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 1441.514559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1441.580403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1441.628300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1441.665405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1441.702748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1441.768447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1441.859112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.26@tcp [ 1441.902489] LNetError: 68832:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.26@tcp to recovery [ 1442.401826] LNet: There was an unexpected network error while writing to 192.168.202.26: rc = -22 [ 1445.692891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1448.544556] LNet: There was an unexpected network error while writing to 192.168.202.26: rc = -22 [ 1448.548725] LNet: Skipped 1 previous similar message [ 1452.979738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1453.024258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1453.078425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1463.149585] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:41:45 (1788831705) [ 1463.350739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.371119] LNet: 69538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.386234] LNetError: 69538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.416300] LNet: Removed LNI 192.168.202.26@tcp [ 1464.444832] LNet: Removed LNI 192.168.202.26@tcp1 [ 1464.556737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.731799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.809413] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1466.266905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1466.327512] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1468.058581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 1468.113160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1468.169914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1468.244082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1468.308262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1468.362948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1468.428049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.26@tcp --health 0 [ 1468.620786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1468.769113] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.26@tcp [ 1469.817411] LNet: There was an unexpected network error while writing to 192.168.202.26: rc = -22 [ 1469.837693] LNet: Skipped 1 previous similar message [ 1483.407576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1483.560171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1483.611727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1483.654799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.26@tcp --health 500 [ 1483.747236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1489.539607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1489.631421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1489.654494] LNet: 70017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.676811] LNetError: 70017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.696800] LNetError: 70017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1489.709403] LNet: Removed LNI 192.168.202.26@tcp [ 1489.742639] LNet: Removed LNI 192.168.202.26@tcp1 [ 1489.775286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1489.874603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1489.918219] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1491.302226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1491.332120] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1492.544653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 1492.603099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1492.659298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1492.684575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1492.728962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1492.761890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1492.800110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1492.851770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.26@tcp --health 0 [ 1493.920410] LNet: There was an unexpected network error while writing to 192.168.202.26: rc = -22 [ 1493.924754] LNet: Skipped 3 previous similar messages [ 1494.364144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1503.860459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1503.886924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1503.914715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1512.952735] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:42:35 (1788831755) [ 1513.538438] LNet: 71055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.548379] LNetError: 71055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.560717] LNetError: 71055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1513.571411] LNet: Removed LNI 192.168.202.26@tcp [ 1513.584709] LNet: Removed LNI 192.168.202.26@tcp1 [ 1514.860803] Key type .llcrypt unregistered [ 1514.863260] Key type ._llcrypt unregistered [ 1522.106889] Key type ._llcrypt registered [ 1522.108827] Key type .llcrypt registered [ 1522.458791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1522.476980] alg: No test for adler32 (adler32-zlib) [ 1523.434302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1523.470371] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1533.123927] Lustre: DEBUG MARKER: Initial discovery [ 1533.189309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.126@tcp [ 1535.986411] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1536.264949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1536.296995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.126@tcp [ 1536.321701] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1536.354735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1536.384860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.126@tcp -r 1 -m GET -l 3 [ 1536.425474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.126@tcp -r 1 -m GET -e local_error [ 1536.467925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.126@tcp -r 1 -m PUT -l 6 [ 1536.800139] LNet: 1 local NIs in recovery (showing 1): 192.168.202.26@tcp [ 1538.123989] Lustre: DEBUG MARKER: Force oleg226-client.virtnet to discover 192.168.202.126@tcp (in background) [ 1538.222100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.126@tcp [ 1539.861376] Lustre: DEBUG MARKER: Force oleg226-server to discover 192.168.202.26@tcp [ 1541.285725] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1542.939310] Lustre: DEBUG MARKER: Wait for 71806 [ 1544.827778] Lustre: DEBUG MARKER: Finished wait on 71806 [ 1544.900266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1544.960541] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1544.979611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1545.338395] LNet: 72265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.349602] LNetError: 72265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.410236] LNet: Removed LNI 192.168.202.26@tcp [ 1547.047649] Key type .llcrypt unregistered [ 1547.054863] Key type ._llcrypt unregistered [ 1565.884056] Key type ._llcrypt registered [ 1565.885682] Key type .llcrypt registered [ 1567.534628] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:43:30 (1788831810) [ 1568.162908] Key type .llcrypt unregistered [ 1568.166624] Key type ._llcrypt unregistered [ 1574.081734] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1574.270970] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1574.278517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1574.291305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1576.909317] Key type ._llcrypt registered [ 1576.911160] Key type .llcrypt registered [ 1577.078827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1577.105167] alg: No test for adler32 (adler32-zlib) [ 1577.991605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.049710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.089427] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1578.833833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1578.852872] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1586.598994] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:43:49 (1788831829) [ 1586.795922] LNet: 74348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.804209] LNetError: 74348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1586.818738] LNet: Removed LNI 10.1.2.3@tcp [ 1586.839956] LNet: Removed LNI 192.168.202.26@tcp [ 1587.519451] Key type .llcrypt unregistered [ 1587.525850] Key type ._llcrypt unregistered [ 1594.005920] Key type ._llcrypt registered [ 1594.008088] Key type .llcrypt registered [ 1594.166781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1594.179661] alg: No test for adler32 (adler32-zlib) [ 1595.052971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.112282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1595.158317] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1596.317717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1596.342783] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1597.702709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.26@tcp [ 1598.752152] LNetError: Unexpected error -22 connecting to 192.168.202.26@tcp at host 192.168.202.26:988 [ 1607.929339] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:44:09 (1788831849) [ 1608.018917] LNet: 75475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.024902] LNetError: 75475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1609.056361] LNet: Removed LNI 10.1.2.3@tcp [ 1609.060591] LNetError: 75475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.078589] LNet: Removed LNI 192.168.202.26@tcp [ 1609.940665] Key type .llcrypt unregistered [ 1609.943076] Key type ._llcrypt unregistered [ 1617.997566] Key type ._llcrypt registered [ 1618.003526] Key type .llcrypt registered [ 1618.111638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1618.175432] alg: No test for adler32 (adler32-zlib) [ 1619.132455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.218518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1619.267150] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1620.157625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1620.199149] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 1621.708951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.26@tcp1 --nid 192.168.202.26@tcp2 [ 1627.198774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.26@tcp1 192.168.202.26@tcp2 [ 1627.234137] LNetError: 76314:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.26@tcp1 to 192.168.202.26@tcp2 - Route table may be misconfigured [ 1627.278735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.26@tcp2 192.168.202.26@tcp1 [ 1635.664953] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:44:38 (1788831878) [ 1635.953563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.015324] LNet: 76888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.037257] LNetError: 76888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.088398] LNet: Removed LNI 192.168.202.26@tcp1 [ 1637.090655] LNetError: 76888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.116593] LNet: Removed LNI 192.168.202.26@tcp2 [ 1637.153457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.226663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1637.269789] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1638.173614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1638.226145] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1639.072758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 1639.111173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.26@tcp 192.168.202.26@tcp [ 1639.153205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.26@tcp1 192.168.202.26@tcp1 [ 1639.213918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.26@tcp -d 192.168.202.26@tcp -e network_timeout [ 1639.245546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -e network_timeout [ 1639.276260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -e network_timeout [ 1639.311157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -e network_timeout [ 1639.354219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1639.374263] LNet: There was an unexpected network error while writing to 192.168.202.26: rc = -22 [ 1639.393978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1639.507555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1639.625922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1640.288586] LNet: 1 local NIs in recovery (showing 1): 192.168.202.26@tcp [ 1645.461267] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:44:48 (1788831888) [ 1645.795854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.848777] LNet: 77750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.853867] LNetError: 77750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.862512] LNet: Removed LNI 192.168.202.26@tcp [ 1646.890047] LNet: Removed LNI 192.168.202.26@tcp1 [ 1646.941077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.014660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1647.283992] LNet: 77787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.694741] Key type .llcrypt unregistered [ 1647.697553] Key type ._llcrypt unregistered [ 1659.666341] Key type ._llcrypt registered [ 1659.669155] Key type .llcrypt registered [ 1661.093109] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:45:03 (1788831903) [ 1661.452537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1661.477323] alg: No test for adler32 (adler32-zlib) [ 1662.437750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.806788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1666.201198] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:45:08 (1788831908) [ 1666.288271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.321203] LNet: 78879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.338116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.444238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1666.467861] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1667.436609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1667.456958] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1668.354794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1668.396247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp1 [ 1668.443477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp1 [ 1675.741216] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:45:18 (1788831918) [ 1677.970283] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg226-server [ 1679.698567] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:45:22 (1788831922) [ 1681.851755] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg226-server [ 1684.262746] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:45:26 (1788831926) [ 1686.469205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg226-server [ 1688.252262] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:45:31 (1788831931) [ 1690.483362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg226-server [ 1692.796984] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:45:35 (1788831935) [ 1694.752979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg226-server [ 1697.185909] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:45:39 (1788831939) [ 1699.208656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg226-server [ 1700.973323] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:45:43 (1788831943) [ 1702.939626] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg226-server [ 1704.450510] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:45:47 (1788831947) [ 1706.603653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg226-server [ 1708.525925] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:45:51 (1788831951) [ 1710.899756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg226-server [ 1712.937299] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:45:55 (1788831955) [ 1713.246793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.277290] LNet: 82761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.286043] LNetError: 82761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.295557] LNet: Removed LNI 192.168.202.26@tcp [ 1714.345214] LNet: Removed LNI 192.168.202.26@tcp1 [ 1714.411442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.505789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.557064] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1715.454305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1715.537985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1715.730601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.752646] LNet: 82909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.758670] LNetError: 82909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.763639] LNetError: 82909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1715.779209] LNet: Removed LNI 192.168.202.26@tcp [ 1715.796317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.864620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1715.885459] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1716.969416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1717.171514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.194310] LNetError: 83054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.212292] LNet: Removed LNI 192.168.202.26@tcp [ 1717.228509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.291301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.332406] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1718.178527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1718.346191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1718.571575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.605541] LNet: 83201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.609857] LNet: 83201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1719.654994] LNet: Removed LNI 192.168.202.26@tcp [ 1719.705440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.789715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1719.837240] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1720.680722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1720.844700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.874484] LNetError: 83347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.879263] LNetError: 83347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1720.889763] LNet: Removed LNI 192.168.202.26@tcp [ 1720.904805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.958567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1720.989830] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1721.715887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1721.824732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1722.096440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.196306] LNet: Removed LNI 192.168.202.26@tcp [ 1723.232906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.290183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1723.316737] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1724.309411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1724.520944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.548132] LNet: 83639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.553435] LNet: 83639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1725.616863] LNet: Removed LNI 192.168.202.26@tcp [ 1725.668945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.750032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.782336] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1726.446585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1726.516368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1726.706298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.726784] LNetError: 83787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.732699] LNetError: 83787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1726.748454] LNet: Removed LNI 192.168.202.26@tcp [ 1726.766763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.823211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1726.852276] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1727.612955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1727.860941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.934362] LNet: Removed LNI 192.168.202.26@tcp [ 1728.985627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.068227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.113758] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1730.022687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1730.160503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1730.480449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.570068] LNet: Removed LNI 192.168.202.26@tcp [ 1731.602940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.730607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1731.786380] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1732.963499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1733.299993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.327171] LNet: 84225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.337449] LNet: 84225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1734.375241] LNet: Removed LNI 192.168.202.26@tcp [ 1734.426653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.538172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1734.601502] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1735.663947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1735.791645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1737.437920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.466758] LNetError: 84382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.472516] LNetError: 84382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1737.490585] LNet: Removed LNI 192.168.202.26@tcp [ 1737.520054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.588399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1737.619595] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1738.418688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 1738.609421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.691667] LNet: Removed LNI 192.168.202.26@tcp [ 1739.750863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.791693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.812063] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1740.509898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1740.553214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.576338] LNet: Removed LNI 192.168.202.26@tcp [ 1740.589903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.631751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1740.684537] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1741.723269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1741.788139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.828996] LNet: Removed LNI 192.168.202.26@tcp [ 1741.851982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.921686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1742.021573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.068641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.122549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1742.237499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.292241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.384492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1742.527900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.582252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.659607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1742.728988] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1743.612759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.638949] LNet: Removed LNI 192.168.202.26@tcp [ 1743.682806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.777894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1751.917165] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:46:34 (1788831994) [ 1752.162483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.202705] LNet: 85550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.207299] LNet: 85550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1752.224802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.282724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1752.330941] LNet: Added LNI 192.168.202.26@tcp231 [8/256/0/180] [ 1753.766905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.806796] LNetError: 85688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.812701] LNetError: 85688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1753.831329] LNet: Removed LNI 192.168.202.26@tcp231 [ 1753.853870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.928425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1753.959918] LNet: Added LNI 192.168.202.26@tcp231 [8/256/0/180] [ 1753.964240] LNet: 85696:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1754.666991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1754.708682] LNet: Removed LNI 192.168.202.26@tcp231 [ 1754.749822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1754.798564] LNet: Added LNI 192.168.202.26@tcp231 [8/256/0/180] [ 1756.127428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1756.190219] LNet: Removed LNI 192.168.202.26@tcp231 [ 1756.212197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.300488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1756.343255] LNet: Added LNI 192.168.202.26@tcp231 [8/256/0/0] [ 1756.362984] LNet: 85848:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1756.909730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1756.968334] LNet: Removed LNI 192.168.202.26@tcp231 [ 1756.979702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1757.022235] LNet: Added LNI 192.168.202.26@tcp231 [8/256/0/0] [ 1758.256561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.301322] LNet: Removed LNI 192.168.202.26@tcp231 [ 1758.336663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.426356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1758.453248] LNet: Added LNI 192.168.202.26@tcp231 [8/256/0/60] [ 1758.460777] LNet: 86000:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1758.986625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1759.011393] LNet: Removed LNI 192.168.202.26@tcp231 [ 1759.024887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1759.057438] LNet: Added LNI 192.168.202.26@tcp231 [8/256/0/60] [ 1760.084173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.128366] LNet: Removed LNI 192.168.202.26@tcp231 [ 1760.144134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.187853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1760.207689] LNet: Added LNI 192.168.202.26@tcp231 [8/256/0/180] [ 1760.694754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1760.725725] LNet: Removed LNI 192.168.202.26@tcp231 [ 1760.743054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1760.767180] LNet: Added LNI 192.168.202.26@tcp231 [8/256/0/180] [ 1768.127423] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:46:51 (1788832011) [ 1768.240686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.305355] LNet: Removed LNI 192.168.202.26@tcp231 [ 1768.357027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.461755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1768.487925] LNet: Added LNI 192.168.202.26@tcp232 [8/256/0/180] [ 1769.348289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1769.411028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1769.465324] LNet: Removed LNI 192.168.202.26@tcp232 [ 1769.486154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1769.532470] LNet: Added LNI 192.168.202.26@tcp232 [8/256/0/180] [ 1770.107394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1770.177335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1770.224371] LNet: Removed LNI 192.168.202.26@tcp232 [ 1770.260506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1770.297808] LNet: Added LNI 192.168.202.26@tcp232 [8/256/0/180] [ 1771.072887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1771.117884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1771.146754] LNet: Removed LNI 192.168.202.26@tcp232 [ 1771.176543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1771.219952] LNet: Added LNI 192.168.202.26@tcp232 [8/256/0/180] [ 1772.044303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1772.104882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1772.141575] LNet: Removed LNI 192.168.202.26@tcp232 [ 1772.170099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1779.074781] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:47:02 (1788832022) [ 1780.510482] Key type .llcrypt unregistered [ 1780.513319] Key type ._llcrypt unregistered [ 1786.728685] Key type ._llcrypt registered [ 1786.730177] Key type .llcrypt registered [ 1786.998697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1787.030703] alg: No test for adler32 (adler32-zlib) [ 1788.020285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1788.055699] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1795.633920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 1804.013606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.067000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1804.104473] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1805.501737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_interrupt -r 2 [ 1805.542907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_interrupt -r 2 [ 1805.578771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_interrupt -r 2 [ 1805.615685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_interrupt -r 2 [ 1805.645478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_interrupt -r 2 [ 1805.681584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_interrupt -r 2 [ 1805.740466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_interrupt -r 2 [ 1805.801978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_interrupt -r 2 [ 1805.831734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.26@tcp 192.168.202.126@tcp [ 1805.853391] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1805.857099] LNet: 1 local NIs in recovery (showing 1): 192.168.202.26@tcp [ 1806.080181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1806.106030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1806.130663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1806.153204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1806.176312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1806.208506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1806.234132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1806.257937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1806.271930] LNet: 88649:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1806.285477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1806.317974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1806.346898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1806.374347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1806.411734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1806.435471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1806.936588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_dropped -r 2 [ 1806.965699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_dropped -r 2 [ 1806.995090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_dropped -r 2 [ 1807.018286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_dropped -r 2 [ 1807.050716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_dropped -r 2 [ 1807.088217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_dropped -r 2 [ 1807.125341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_dropped -r 2 [ 1807.177399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_dropped -r 2 [ 1807.214922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.26@tcp 192.168.202.126@tcp [ 1807.236273] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1807.443489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1807.476364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1807.495920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1807.517447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1807.541669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1807.562852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1807.585738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1807.608062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1807.621760] LNet: 88770:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1807.636867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1807.659362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1807.678919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1807.706126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1807.730882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1807.759623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1808.206121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_aborted -r 2 [ 1808.226803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_aborted -r 2 [ 1808.269574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_aborted -r 2 [ 1808.302771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_aborted -r 2 [ 1808.328399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_aborted -r 2 [ 1808.354922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_aborted -r 2 [ 1808.408469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_aborted -r 2 [ 1808.436104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_aborted -r 2 [ 1808.457419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.26@tcp 192.168.202.126@tcp [ 1808.474195] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1808.706750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1808.728884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1808.753787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1808.775205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1808.804274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1808.841420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1808.895590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1808.930285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1808.951233] LNet: 88890:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1808.978389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1809.007397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1809.039786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1809.080630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1809.119575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1809.186632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1809.711406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_no_route -r 2 [ 1809.766945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_no_route -r 2 [ 1809.804183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_no_route -r 2 [ 1809.860946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_no_route -r 2 [ 1809.927950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_no_route -r 2 [ 1810.002794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_no_route -r 2 [ 1810.065488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_no_route -r 2 [ 1810.095724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_no_route -r 2 [ 1810.123076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.26@tcp 192.168.202.126@tcp [ 1810.397780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1810.423230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1810.440902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1810.466719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1810.487351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1810.511693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1810.530839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1810.546966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1810.563695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1810.584564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1810.611509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1810.633789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1810.657216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1810.684250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1811.232846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e local_timeout -r 2 [ 1811.273318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e local_timeout -r 2 [ 1811.310835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e local_timeout -r 2 [ 1811.354757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e local_timeout -r 2 [ 1811.391618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e local_timeout -r 2 [ 1811.446563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e local_timeout -r 2 [ 1811.486322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e local_timeout -r 2 [ 1811.538212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e local_timeout -r 2 [ 1811.584776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.26@tcp 192.168.202.126@tcp [ 1811.610588] LNet: There was an unexpected network error while writing to 192.168.202.126: rc = -22 [ 1811.615544] LNet: Skipped 1 previous similar message [ 1811.994700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1812.025826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1812.054823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1812.098310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1812.157846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1812.195358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1812.220699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1812.273815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1812.311524] LNet: 89131:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1812.317124] LNet: 89131:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1812.337771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1812.370885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1812.421989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1812.467404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1812.494364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1812.541682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1813.069257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp -m GET -e remote_dropped -r 2 [ 1813.109799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.126@tcp1 -m GET -e remote_dropped -r 2 [ 1813.148397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp -m GET -e remote_dropped -r 2 [ 1813.184564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp -d 192.168.202.26@tcp1 -m GET -e remote_dropped -r 2 [ 1813.219835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp -m GET -e remote_dropped -r 2 [ 1813.253748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.26@tcp1 -d 192.168.202.126@tcp1 -m GET -e remote_dropped -r 2 [ 1813.285818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp -m GET -e remote_dropped -r 2 [ 1813.320637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.26@tcp1 -d 192.168.202.26@tcp1 -m GET -e remote_dropped -r 2 [ 1813.356804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.26@tcp 192.168.202.126@tcp [ 1813.372948] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.126@tcp [ 1813.674845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1813.719586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1813.756967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1813.787726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1813.845746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1813.889556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1813.916267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1813.941840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1813.979797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1814.011524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1814.056605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1814.118285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1814.170606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1814.227859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1819.200312] LNet: 89325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.204990] LNetError: 89325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.211558] LNet: Removed LNI 192.168.202.26@tcp [ 1819.233207] LNet: Removed LNI 192.168.202.26@tcp1 [ 1819.628675] Key type .llcrypt unregistered [ 1819.630555] Key type ._llcrypt unregistered [ 1833.934799] Key type ._llcrypt registered [ 1833.936578] Key type .llcrypt registered [ 1834.955753] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:47:58 (1788832078) [ 1835.474748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1835.495060] alg: No test for adler32 (adler32-zlib) [ 1836.455913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.507947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1836.541053] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1837.301697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1837.341766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.26@tcp [ 1837.362944] LNetError: 90302:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.26@tcp to recovery [ 1837.442774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.26@tcp [ 1837.559874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1843.220866] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:48:06 (1788832086) [ 1843.533527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1843.552946] LNet: 90891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.558475] LNetError: 90891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.572416] LNet: Removed LNI 192.168.202.26@tcp [ 1843.589586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.638954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1843.666750] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1844.389932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1844.399867] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 1845.548804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.26@tcp [ 1845.603369] LNetError: 91157:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.26@tcp to recovery [ 1845.667333] LNet: 1 local NIs in recovery (showing 1): 192.168.202.26@tcp [ 1845.690427] LNet: There was an unexpected network error while writing to 192.168.202.26: rc = -22 [ 1847.069838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.26@tcp [ 1855.748233] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:48:18 (1788832098) [ 1857.799540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg226-server [ 1859.471904] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:48:22 (1788832102) [ 1861.663453] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg226-server [ 1863.205457] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:48:26 (1788832106) [ 1864.953853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg226-server [ 1866.684623] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:48:29 (1788832109) [ 1866.843493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.879925] LNet: 92776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.898428] LNetError: 92776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.936480] LNet: Removed LNI 192.168.202.26@tcp [ 1867.939314] LNetError: 92776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.950398] LNet: Removed LNI 192.168.202.26@tcp1 [ 1867.995741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.072345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1868.098315] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1868.942133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1868.993494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1869.033787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1869.131486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1869.178700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1869.240831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1875.504705] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:48:38 (1788832118) [ 1875.874862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.918113] LNet: 93512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.930143] LNetError: 93512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.971666] LNet: Removed LNI 192.168.202.26@tcp [ 1876.001476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.129929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1876.159475] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1884.540952] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:48:47 (1788832127) [ 1884.770451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.795278] LNet: 94216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.800430] LNetError: 94216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.816792] LNet: Removed LNI 192.168.202.26@tcp [ 1884.837436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.891671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.913514] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1891.745668] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:48:54 (1788832134) [ 1892.895194] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1894.720548] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:48:57 (1788832137) [ 1895.197817] LNet: 95252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.203329] LNetError: 95252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.217848] LNet: Removed LNI 192.168.202.26@tcp [ 1895.750078] Key type .llcrypt unregistered [ 1895.756653] Key type ._llcrypt unregistered [ 1901.890689] Key type ._llcrypt registered [ 1901.892283] Key type .llcrypt registered [ 1902.110245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1902.121856] alg: No test for adler32 (adler32-zlib) [ 1903.025926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1903.051298] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1910.623582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 1915.623975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.126@tcp [ 1916.133963] LNet: 95667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.147460] LNetError: 95667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.160702] LNet: Removed LNI 192.168.202.26@tcp [ 1916.708587] Key type .llcrypt unregistered [ 1916.710645] Key type ._llcrypt unregistered [ 1928.808788] Key type ._llcrypt registered [ 1928.812955] Key type .llcrypt registered [ 1930.130794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1931.547920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1933.822788] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:49:36 (1788832176) [ 1935.797621] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1937.773843] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:49:40 (1788832180) [ 1940.418528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg226-server [ 1942.308117] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:49:45 (1788832185) [ 1944.336820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg226-server [ 1946.570612] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:49:49 (1788832189) [ 1947.995382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1948.036686] alg: No test for adler32 (adler32-zlib) [ 1949.344583] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1949.775131] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 1951.482044] Key type lgssc registered [ 1953.316240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1976.859271] Key type lgssc unregistered [ 1977.178682] LNet: 98839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.184782] LNetError: 98839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.200883] LNet: Removed LNI 192.168.202.26@tcp [ 1978.090990] Key type .llcrypt unregistered [ 1978.102825] Key type ._llcrypt unregistered [ 1994.152373] Key type ._llcrypt registered [ 1994.154015] Key type .llcrypt registered [ 1995.803730] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:50:38 (1788832238) [ 1996.353507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1996.370249] alg: No test for adler32 (adler32-zlib) [ 1997.321897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.409265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1997.480922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1997.527261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1997.576568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2005.523981] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:50:48 (1788832248) [ 2005.608265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.630340] LNet: 100242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.666438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.753513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2005.785174] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2006.965402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2006.995446] LNet: 100395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.001502] LNetError: 100395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.012822] LNet: Removed LNI 192.168.202.26@tcp [ 2007.036936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.114255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2007.154968] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2008.057306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2008.091929] LNet: 100424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.101451] LNetError: 100424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2008.118033] LNet: Removed LNI 192.168.202.26@tcp [ 2008.152615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.253178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2008.300404] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2016.853885] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:50:59 (1788832259) [ 2016.981480] LNet: 101008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.993961] LNetError: 101008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2017.044675] LNet: Removed LNI 192.168.202.26@tcp [ 2018.235893] Key type .llcrypt unregistered [ 2018.240549] Key type ._llcrypt unregistered [ 2025.722910] Key type ._llcrypt registered [ 2025.725279] Key type .llcrypt registered [ 2025.908560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2025.933454] alg: No test for adler32 (adler32-zlib) [ 2027.036621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.183451] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2027.185433] LNet: *** cfs_fail_loc=e003, val=100*** [ 2028.419702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2028.453055] LNet: *** cfs_fail_loc=e003, val=100*** [ 2028.456413] LNet: Skipped 2 previous similar messages [ 2028.461198] Ping reply received with mapping: [ 2028.462593] found metadata reply NI[192.168.202.26@tcp] [ 2028.464278] 1a ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2028.469320] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2028.472044] 5a 5a 5a 5a 5a 5a 5a 5a [ 2028.852590] LNet: 101409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.862411] LNetError: 101409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.892932] LNet: Removed LNI 192.168.202.26@tcp [ 2029.829828] Key type .llcrypt unregistered [ 2029.832921] Key type ._llcrypt unregistered [ 2045.473202] Key type ._llcrypt registered [ 2045.474630] Key type .llcrypt registered [ 2047.955623] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:51:29 (1788832289) [ 2048.681635] Key type .llcrypt unregistered [ 2048.684766] Key type ._llcrypt unregistered [ 2056.471648] Key type ._llcrypt registered [ 2056.474860] Key type .llcrypt registered [ 2056.828950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2056.891327] alg: No test for adler32 (adler32-zlib) [ 2058.158972] LNetError: 102446:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2058.685280] Key type .llcrypt unregistered [ 2058.687157] Key type ._llcrypt unregistered [ 2064.643104] Key type ._llcrypt registered [ 2064.649478] Key type .llcrypt registered [ 2065.998320] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:51:48 (1788832308) [ 2066.809474] Key type .llcrypt unregistered [ 2066.813117] Key type ._llcrypt unregistered [ 2074.622922] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2074.627933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2077.604428] Key type ._llcrypt registered [ 2077.607329] Key type .llcrypt registered [ 2077.870496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2077.896777] alg: No test for adler32 (adler32-zlib) [ 2078.922781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.025577] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2088.504063] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:52:11 (1788832331) [ 2088.682207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.699990] LNet: 104220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.705621] LNetError: 104220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2088.717563] LNet: Removed LNI 10.1.2.3@tcp [ 2088.754310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.622350] Key type .llcrypt unregistered [ 2089.626408] Key type ._llcrypt unregistered [ 2096.298934] Key type ._llcrypt registered [ 2096.300514] Key type .llcrypt registered [ 2096.627888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2096.656410] alg: No test for adler32 (adler32-zlib) [ 2097.626925] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2097.704025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2097.863730] LNet: 104505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.870606] LNetError: 104505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.926175] LNet: Removed LNI 192.168.202.26@tcp [ 2099.852088] Key type .llcrypt unregistered [ 2099.855836] Key type ._llcrypt unregistered [ 2112.390925] Key type ._llcrypt registered [ 2112.394737] Key type .llcrypt registered [ 2113.879538] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:52:36 (1788832356) [ 2114.924433] Key type .llcrypt unregistered [ 2114.927199] Key type ._llcrypt unregistered [ 2122.060123] Key type ._llcrypt registered [ 2122.062151] Key type .llcrypt registered [ 2122.338607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2122.358310] alg: No test for adler32 (adler32-zlib) [ 2130.220879] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:52:53 (1788832373) [ 2130.348661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2130.420795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.513107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2130.596089] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2131.423334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2131.457418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2131.485941] LNet: Removed LNI 192.168.202.26@tcp [ 2131.761496] LNet: 106244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.200403] Key type .llcrypt unregistered [ 2132.203456] Key type ._llcrypt unregistered [ 2145.594427] Key type ._llcrypt registered [ 2145.596830] Key type .llcrypt registered [ 2147.123636] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:53:09 (1788832389) [ 2147.542701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2147.554712] alg: No test for adler32 (adler32-zlib) [ 2148.534561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.589995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2148.613209] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2149.329565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2149.374712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.26@tcp [ 2149.397683] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2149.402847] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2149.406717] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2149, 0, 0 [ 2149.413607] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2149.416781] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2149.422494] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2149.429193] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2149, 0, 0 [ 2149.435892] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2149.441238] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2149.447439] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2149.454339] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2149, 0, 2199 [ 2149.461647] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2149.467303] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2149.476228] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2149.479716] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2149, 0, 2199 [ 2149.489572] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2157.166733] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:53:19 (1788832399) [ 2157.621228] LNet: 107781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.625728] LNetError: 107781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2157.642604] LNet: Removed LNI 192.168.202.26@tcp [ 2158.308381] Key type .llcrypt unregistered [ 2158.309639] Key type ._llcrypt unregistered [ 2164.418504] Key type ._llcrypt registered [ 2164.421328] Key type .llcrypt registered [ 2164.575363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2164.588670] alg: No test for adler32 (adler32-zlib) [ 2165.610639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2165.686936] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2173.000306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 2181.662824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.740139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2181.779321] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 2183.106283] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2183.378724] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2183.383080] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2183.387332] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2185.826804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2185.864586] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2198.800953] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:54:01 (1788832441) [ 2198.916943] LNet: 109277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.926679] LNetError: 109277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.970369] LNet: Removed LNI 192.168.202.26@tcp [ 2199.973387] LNetError: 109277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.989858] LNet: Removed LNI 192.168.202.26@tcp1 [ 2200.004542] LNet: Removed LNI 10.1.2.3@tcp99 [ 2200.854975] Key type .llcrypt unregistered [ 2200.861960] Key type ._llcrypt unregistered [ 2208.686210] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2208.697164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2211.292779] Key type ._llcrypt registered [ 2211.298200] Key type .llcrypt registered [ 2211.509860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2211.534373] alg: No test for adler32 (adler32-zlib) [ 2212.453852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.536886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.565548] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2213.169790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2213.189109] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2215.194777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2215.241620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2215.301563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2215.324673] LNetError: 110026:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2215.365445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2215.418661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2215.970064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2216.041213] LNet: 110046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.046546] LNetError: 110046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2216.053748] LNet: Removed LNI 10.1.2.3@tcp [ 2217.127956] LNet: Removed LNI 192.168.202.26@tcp [ 2217.713748] Key type .llcrypt unregistered [ 2217.715769] Key type ._llcrypt unregistered [ 2230.377399] Key type ._llcrypt registered [ 2230.382252] Key type .llcrypt registered [ 2232.015377] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:54:34 (1788832474) [ 2232.524484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.549810] alg: No test for adler32 (adler32-zlib) [ 2233.524925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.582118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2233.626614] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2234.404308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2234.472932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg226-client@tcp [ 2242.650793] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:54:45 (1788832485) [ 2243.088149] LNet: 111577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.108174] LNetError: 111577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2243.137931] LNet: Removed LNI 192.168.202.26@tcp [ 2243.861516] Key type .llcrypt unregistered [ 2243.869510] Key type ._llcrypt unregistered [ 2251.393332] Key type ._llcrypt registered [ 2251.395808] Key type .llcrypt registered [ 2251.581654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2251.621614] alg: No test for adler32 (adler32-zlib) [ 2252.577993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2252.606437] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2252.650741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2259.131545] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:55:02 (1788832502) [ 2260.956432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2262.852947] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:55:05 (1788832505) [ 2264.425823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2266.261594] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:55:09 (1788832509) [ 2266.434600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2266.457201] LNet: 112998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2266.464633] LNetError: 112998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.481342] LNet: Removed LNI 192.168.202.26@tcp [ 2266.504040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.588498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2266.624181] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2267.316959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2267.377115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2274.052888] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:55:17 (1788832517) [ 2274.227903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2274.257527] LNet: 113707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.265565] LNetError: 113707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.284823] LNet: Removed LNI 192.168.202.26@tcp [ 2274.302492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.359360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2274.386900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2274.900680] Key type .llcrypt unregistered [ 2274.902586] Key type ._llcrypt unregistered [ 2287.174677] Key type ._llcrypt registered [ 2287.176444] Key type .llcrypt registered [ 2288.567810] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:55:31 (1788832531) [ 2289.089514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2289.108644] alg: No test for adler32 (adler32-zlib) [ 2290.055722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.102694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2290.128600] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2290.792329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2290.847399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2297.580305] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:55:40 (1788832540) [ 2297.685177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2297.731649] LNet: 115256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.741471] LNetError: 115256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.773473] LNet: Removed LNI 192.168.202.26@tcp [ 2297.837994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.890758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2297.932293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2304.445842] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:55:47 (1788832547) [ 2304.849324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.876279] LNet: 115840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.900805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.977243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2305.368460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2305.405940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2305.482394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2305.978519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2306.023472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2306.112385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.145240] LNet: 115873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.173750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.252737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2306.615483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2306.646379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2306.697274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2307.111940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2307.136162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2313.703269] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:55:56 (1788832556) [ 2314.052836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2314.085922] LNet: 116474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2314.138496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.241087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2314.289351] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2315.059808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2315.076982] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2315.737424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2315.800754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2316.047995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.077681] LNetError: 116753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.088964] LNet: Removed LNI 192.168.202.26@tcp [ 2317.095143] LNetError: 116753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.120868] LNet: Removed LNI 192.168.202.26@tcp2 [ 2317.201816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.319584] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2317.327791] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2317.333154] LNet: 116759:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2317.699036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2318.033695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2318.092506] LNet: 116796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.104122] LNet: 116796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2319.136258] LNet: Removed LNI 192.168.202.26@tcp [ 2319.140424] LNetError: 116796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2319.155711] LNetError: 116796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2319.177859] LNet: Removed LNI 192.168.202.26@tcp2 [ 2319.216863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.295949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.336915] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2320.025322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2320.053814] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2320.828807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2320.858780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2320.917357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2321.142518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.208683] LNet: Removed LNI 192.168.202.26@tcp [ 2323.242361] LNet: Removed LNI 192.168.202.26@tcp2 [ 2323.289481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.358699] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2323.364320] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2323.368522] LNet: 117087:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2324.020757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2324.235896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.255939] LNetError: 117124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.262511] LNetError: 117124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2324.269921] LNet: Removed LNI 192.168.202.26@tcp [ 2325.293619] LNet: Removed LNI 192.168.202.26@tcp2 [ 2325.327634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.421089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2325.458543] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2326.108880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2326.142649] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2326.961445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2327.031330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2327.284482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.313174] LNet: 117404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.321271] LNet: 117404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2328.355486] LNet: Removed LNI 192.168.202.26@tcp [ 2328.367446] LNet: Removed LNI 192.168.202.26@tcp2 [ 2328.395215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.500591] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2328.514386] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2328.524995] LNet: 117410:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2329.013360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2329.281180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.308620] LNet: Removed LNI 192.168.202.26@tcp [ 2329.355495] LNet: Removed LNI 192.168.202.26@tcp2 [ 2329.427530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.524551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2329.561177] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2330.352305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2330.381494] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2331.364795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2331.417743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2331.503380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2331.722286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.768248] LNet: Removed LNI 192.168.202.26@tcp [ 2332.781140] LNetError: 117731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.791283] LNetError: 117731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2332.805091] LNet: Removed LNI 192.168.202.26@tcp2 [ 2332.851091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.903761] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2332.909827] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2332.912926] LNet: 117737:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2333.526159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2333.688688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.711649] LNet: Removed LNI 192.168.202.26@tcp [ 2334.775167] LNet: Removed LNI 192.168.202.26@tcp2 [ 2334.806810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.836479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2334.856305] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2335.383696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2335.395703] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2336.239367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2336.273426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2336.337505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2336.649240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.696259] LNet: Removed LNI 192.168.202.26@tcp [ 2337.704790] LNet: Removed LNI 192.168.202.26@tcp2 [ 2337.729108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.766901] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2337.771388] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2337.775678] LNet: 118077:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2338.087922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2338.290622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.360165] LNet: Removed LNI 192.168.202.26@tcp [ 2339.370340] LNet: Removed LNI 192.168.202.26@tcp2 [ 2339.405518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.483880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2339.499926] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2340.177330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2340.193902] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2341.102778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2341.178627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2341.209561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2341.286051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2341.597693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.624849] LNet: Removed LNI 192.168.202.26@tcp [ 2343.655382] LNet: Removed LNI 192.168.202.26@tcp2 [ 2343.711159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.833144] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2343.839143] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2344.672930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2353.847108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2355.683988] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:56:38 (1788832598) [ 2355.951406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2355.983832] LNet: 119197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.991387] LNet: 119197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2355.998059] LNetError: 119197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.005858] LNetError: 119197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 2356.012811] LNet: Removed LNI 192.168.202.26@tcp [ 2357.041525] LNet: Removed LNI 192.168.202.26@tcp2 [ 2357.063895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.329256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2357.375389] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2357.945842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2357.963958] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2359.080605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.120385] LNet: Removed LNI 10.1.2.3@tcp [ 2359.134318] LNet: Removed LNI 192.168.202.26@tcp [ 2359.164334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.227530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2359.284686] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2359.292727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2359.951995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2359.987854] LNet: Removed LNI 192.168.202.26@tcp [ 2360.001282] LNet: Removed LNI 10.1.2.3@tcp [ 2367.388376] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 21:56:50 (1788832610) [ 2367.519239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2367.555624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.601859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2367.641397] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2368.414274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2368.463963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2368.521837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2368.558748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2368.603172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2368.643956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2368.684407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2368.768446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2368.815619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2368.875039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2368.942316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.26@tcp [ 2369.144733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2376.921596] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:56:59 (1788832619) [ 2377.132427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2378.219884] LNet: Removed LNI 192.168.202.26@tcp [ 2378.263068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.433347] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2378.540796] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2378.551555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2378.554838] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2381.039551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2381.075336] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2381.852957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2381.877563] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2384.234180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2384.287864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2384.346587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.26@tcp -m PUT -e local_timeout -r 1 [ 2384.403026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2384.468115] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2384.482581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2384.871757] LNetError: 120878:0:(lib-move.c:4801:lnet_parse()) 192.168.202.26@tcp, src 192.168.202.26@tcp: Dropping PUT (error -108 looking up sender) [ 2385.901720] LNet: Removed LNI 10.1.2.3@tcp [ 2386.926418] LNet: Removed LNI 192.168.202.26@tcp [ 2387.368138] Key type .llcrypt unregistered [ 2387.369268] Key type ._llcrypt unregistered [ 2399.149595] Key type ._llcrypt registered [ 2399.152681] Key type .llcrypt registered [ 2400.172678] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:57:23 (1788832643) [ 2400.509235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2400.537303] alg: No test for adler32 (adler32-zlib) [ 2401.526579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.836951] LNet: 122186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2402.445957] Key type .llcrypt unregistered [ 2402.453283] Key type ._llcrypt unregistered [ 2408.919437] Key type ._llcrypt registered [ 2408.923611] Key type .llcrypt registered [ 2409.288626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2409.313409] alg: No test for adler32 (adler32-zlib) [ 2410.290283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2410.332147] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2419.141390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 2419.334230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2419.875244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.126@tcp [ 2420.206525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2420.246903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2420.292085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2420.319169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2420.351198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2420.380848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2420.408532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2420.434448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2420.447787] LNet: 122632:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2420.464220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2420.513854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2420.546586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2420.578411] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.126@tcp [ 2420.584717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2420.613920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2420.641791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2421.397224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2421.618583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2421.646262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2421.692641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2421.731986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2421.765507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2421.806413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2421.839660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2421.871339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2421.889778] LNet: 122750:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2421.914250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2421.945441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2421.987996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2422.039731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2422.099765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2422.153433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2422.983528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.126@tcp [ 2423.294075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2423.329966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2423.354336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2423.384168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2423.425040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2423.447279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2423.466776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2423.503731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2423.544338] LNet: 122867:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2423.587534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2423.622544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2423.669753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2423.717808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2423.772757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2423.823518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2424.564359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2424.820864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2424.853214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2424.925930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2424.979101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2425.025824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2425.058547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2425.117251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2425.150440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2425.190806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2425.235904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2425.284736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2425.327107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2425.352542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2425.416970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2425.781864] LNet: 123030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.787890] LNetError: 123030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.801775] LNet: Removed LNI 192.168.202.26@tcp [ 2426.244824] Key type .llcrypt unregistered [ 2426.249217] Key type ._llcrypt unregistered [ 2443.199319] Key type ._llcrypt registered [ 2443.201586] Key type .llcrypt registered [ 2444.445341] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:58:07 (1788832687) [ 2445.096377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2445.108128] alg: No test for adler32 (adler32-zlib) [ 2446.151240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.839451] LNet: 123918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.734348] Key type .llcrypt unregistered [ 2447.738465] Key type ._llcrypt unregistered [ 2455.323531] Key type ._llcrypt registered [ 2455.326586] Key type .llcrypt registered [ 2455.528358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2455.535402] alg: No test for adler32 (adler32-zlib) [ 2456.536885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2456.581733] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2466.135214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.126@tcp [ 2477.714797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2477.765061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2477.812683] LNet: Added LNI 192.168.202.26@tcp1 [8/256/0/180] [ 2478.906702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2479.466662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.126@tcp [ 2479.780818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2479.811488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2479.839963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2479.875647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2479.904270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2479.940906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2479.982361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2480.011036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2480.041690] LNet: 124568:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2480.060797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2480.115598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2480.165290] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.126@tcp [ 2480.176448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2480.219895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2480.245838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2480.263193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2481.135718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2481.349722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2481.402540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2481.450057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2481.490966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2481.537608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2481.568502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2481.644953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2481.677792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2481.699180] LNet: 124685:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2481.729545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2481.780888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2481.851865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2481.901792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2481.964493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2482.000282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2482.688464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.126@tcp [ 2483.000573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2483.027156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2483.053955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2483.090325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2483.122392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2483.161643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2483.203114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2483.248238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2483.286324] LNet: 124803:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2483.311957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2483.345259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2483.381656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2483.411521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2483.452867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2483.500221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2484.121436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2484.412592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2484.432576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2484.460545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2484.490480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2484.521249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2484.550991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2484.582536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2484.610528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2484.643451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2484.688467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2484.739340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2484.810462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2484.846227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2484.904723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2490.711086] LNet: 125001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2490.737925] LNetError: 125001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2490.753426] LNet: Removed LNI 192.168.202.26@tcp [ 2490.763074] LNet: Removed LNI 192.168.202.26@tcp1 [ 2491.435815] Key type .llcrypt unregistered [ 2491.439400] Key type ._llcrypt unregistered [ 2509.598558] Key type ._llcrypt registered [ 2509.600861] Key type .llcrypt registered [ 2510.892492] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 21:59:13 (1788832753) [ 2511.384324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.419242] alg: No test for adler32 (adler32-zlib) [ 2512.346793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2512.394954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2512.409982] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2512.772940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2512.788526] LNet: Added LNI 192.168.202.26@tcp2 [8/256/0/180] [ 2514.054276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.26@tcp [ 2514.144645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2515.232313] LNet: Removed LNI 192.168.202.26@tcp [ 2524.786596] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 21:59:26 (1788832766) [ 2527.234191] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2529.357313] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:59:32 (1788832772) [ 2538.669262] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:59:41 (1788832781) [ 2547.008824] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:59:50 (1788832790) [ 2555.470578] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:59:58 (1788832798) [ 2563.231514] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:00:06 (1788832806) [ 2570.783591] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:00:13 (1788832813) [ 2578.446806] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:00:21 (1788832821) [ 2587.674463] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:00:30 (1788832830) [ 2595.238610] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:00:38 (1788832838) [ 2602.414245] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:00:45 (1788832845) [ 2610.850051] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:00:53 (1788832853) [ 2622.306951] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:01:04 (1788832864) [ 2631.178709] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:01:13 (1788832873) [ 2639.187437] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:01:21 (1788832881) [ 2647.521110] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:01:30 (1788832890) [ 2656.560718] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:01:39 (1788832899) [ 2664.890362] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:01:47 (1788832907) [ 2672.850527] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:01:55 (1788832915) [ 2673.050336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2673.087858] LNet: 136705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2673.096942] LNetError: 136705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2673.128322] LNet: Removed LNI 192.168.202.26@tcp2 [ 2673.206770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2673.355122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2673.382380] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2674.789715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2674.816377] LNet: 136853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2674.825391] LNetError: 136853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2674.838359] LNet: Removed LNI 192.168.202.26@tcp [ 2674.863500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2674.932710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2674.960448] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2683.464499] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:02:06 (1788832926) [ 2692.458742] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:02:15 (1788832935) [ 2702.216403] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:02:24 (1788832944) [ 2712.584614] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:02:35 (1788832955) [ 2720.992908] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:02:43 (1788832963) [ 2729.885758] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:02:52 (1788832972) [ 2739.213144] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:03:02 (1788832982) [ 2747.339878] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:03:10 (1788832990) [ 2756.016616] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:03:19 (1788832999) [ 2763.559060] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:03:26 (1788833006) [ 2770.770216] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:03:33 (1788833013) [ 2778.707407] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:03:41 (1788833021) [ 2786.033537] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2525 sec ========== 22:03:49 (1788833029) [ 2786.493461] LNet: 144323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.504540] LNetError: 144323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2786.520665] LNet: Removed LNI 192.168.202.26@tcp [ 2787.239848] Key type .llcrypt unregistered [ 2787.244081] Key type ._llcrypt unregistered [ 2805.287472] Key type ._llcrypt registered [ 2805.327742] Key type .llcrypt registered [ 2805.842503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2805.896208] alg: No test for adler32 (adler32-zlib) [ 2807.195756] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 2807.730966] LNet: Added LNI 192.168.202.26@tcp [8/256/0/180] [ 2809.512243] Key type lgssc registered [ 2811.657185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2889.754691] Lustre: Mounted lustre-client - version 2.17.57_104_ga33798f [ 2895.740766] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2904.070552] Lustre: Unmounted lustre-client [ 2933.984406] Key type lgssc unregistered [ 2934.233427] LNet: 147274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2934.238287] LNetError: 147274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2934.252244] LNet: Removed LNI 192.168.202.26@tcp [ 2934.999276] Key type .llcrypt unregistered [ 2935.001121] Key type ._llcrypt unregistered