[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 361925324 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002334] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007374] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010114] LSM: Security Framework initializing [ 0.011039] Yama: becoming mindful. [ 0.012025] SELinux: Initializing. [ 0.013048] *** VALIDATE selinux *** [ 0.020744] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025713] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026120] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027084] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028093] *** VALIDATE tmpfs *** [ 0.030326] *** VALIDATE proc *** [ 0.031225] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.035145] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036126] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038021] Spectre V2 : User space: Vulnerable [ 0.039004] Speculative Store Bypass: Vulnerable [ 0.041696] debug: unmapping init [mem 0xffffffffa2059000-0xffffffffa2060fff] [ 0.043879] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044617] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045015] ... version: 2 [ 0.046006] ... bit width: 48 [ 0.047006] ... generic registers: 4 [ 0.048007] ... value mask: 0000ffffffffffff [ 0.049007] ... max period: 00007fffffffffff [ 0.050007] ... fixed-purpose events: 3 [ 0.051008] ... event mask: 000000070000000f [ 0.052255] rcu: Hierarchical SRCU implementation. [ 0.054323] smp: Bringing up secondary CPUs ... [ 0.055505] x86: Booting SMP configuration: [ 0.056016] .... node #0, CPUs: #1 #2 #3 [ 0.059015] smp: Brought up 1 node, 4 CPUs [ 0.061009] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.203592] node 0 deferred pages initialised in 140ms [ 0.206177] devtmpfs: initialized [ 0.207158] x86/mm: Memory block size: 128MB [ 0.208784] gcov: version magic: 0x41383552 [ 0.211273] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.212059] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.214275] pinctrl core: initialized pinctrl subsystem [ 0.216145] [ 0.216528] ************************************************************* [ 0.218010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.219005] ** ** [ 0.222009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.224006] ** ** [ 0.226010] ** This means that this kernel is built to expose internal ** [ 0.228009] ** IOMMU data structures, which may compromise security on ** [ 0.230008] ** your system. ** [ 0.233011] ** ** [ 0.236012] ** If you see this message and you are not debugging the ** [ 0.238008] ** kernel, report this immediately to your vendor! ** [ 0.240007] ** ** [ 0.241008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.243008] ************************************************************* [ 0.244485] NET: Registered protocol family 16 [ 0.246373] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.248038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.250031] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.253135] cpuidle: using governor menu [ 0.254294] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.256504] PCI: Using configuration type 1 for base access [ 0.258136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.265097] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.267021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.270107] cryptd: max_cpu_qlen set to 1000 [ 0.272279] ACPI: Added _OSI(Module Device) [ 0.273081] ACPI: Added _OSI(Processor Device) [ 0.274010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.275011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.280078] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.285419] ACPI: Interpreter enabled [ 0.286042] ACPI: PM: (supports S0 S3 S4 S5) [ 0.287009] ACPI: Using IOAPIC for interrupt routing [ 0.289076] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.291329] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.298398] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.300021] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.302011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.305056] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.309063] acpiphp: Slot [2] registered [ 0.309949] acpiphp: Slot [5] registered [ 0.312094] acpiphp: Slot [6] registered [ 0.313092] acpiphp: Slot [3] registered [ 0.314067] acpiphp: Slot [4] registered [ 0.315028] acpiphp: Slot [7] registered [ 0.315862] acpiphp: Slot [8] registered [ 0.317058] acpiphp: Slot [9] registered [ 0.318071] acpiphp: Slot [10] registered [ 0.319082] acpiphp: Slot [11] registered [ 0.320088] acpiphp: Slot [12] registered [ 0.322049] acpiphp: Slot [13] registered [ 0.323055] acpiphp: Slot [14] registered [ 0.324054] acpiphp: Slot [15] registered [ 0.326053] acpiphp: Slot [16] registered [ 0.327088] acpiphp: Slot [17] registered [ 0.328100] acpiphp: Slot [18] registered [ 0.329045] acpiphp: Slot [19] registered [ 0.330061] acpiphp: Slot [20] registered [ 0.331072] acpiphp: Slot [21] registered [ 0.332080] acpiphp: Slot [22] registered [ 0.334054] acpiphp: Slot [23] registered [ 0.335012] acpiphp: Slot [24] registered [ 0.336098] acpiphp: Slot [25] registered [ 0.337049] acpiphp: Slot [26] registered [ 0.337872] acpiphp: Slot [27] registered [ 0.339058] acpiphp: Slot [28] registered [ 0.340065] acpiphp: Slot [29] registered [ 0.341065] acpiphp: Slot [30] registered [ 0.342062] acpiphp: Slot [31] registered [ 0.343063] PCI host bridge to bus 0000:00 [ 0.344011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.347022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.349020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.351027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.353020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.355169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.356918] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360187] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.366354] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.369376] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377520] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379520] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.381031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.383610] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387010] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.394015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402031] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405702] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.421015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.429378] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.437885] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.453015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.469578] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.472388] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.475409] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.478321] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.480177] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.487155] iommu: Default domain type: Passthrough [ 0.492000] SCSI subsystem initialized [ 0.493142] ACPI: bus type USB registered [ 0.495158] usbcore: registered new interface driver usbfs [ 0.497073] usbcore: registered new interface driver hub [ 0.498064] usbcore: registered new device driver usb [ 0.500184] pps_core: LinuxPPS API ver. 1 registered [ 0.502009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.504088] PTP clock support registered [ 0.506094] EDAC MC: Ver: 3.0.0 [ 0.507219] PCI: Using ACPI for IRQ routing [ 0.508821] NetLabel: Initializing [ 0.509011] NetLabel: domain hash size = 128 [ 0.510008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.511108] NetLabel: unlabeled traffic allowed by default [ 0.513164] vgaarb: loaded [ 0.515342] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.517014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.523000] clocksource: Switched to clocksource kvm-clock [ 0.615136] VFS: Disk quotas dquot_6.6.0 [ 0.616507] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.618884] *** VALIDATE ramfs *** [ 0.620018] *** VALIDATE hugetlbfs *** [ 0.622139] pnp: PnP ACPI init [ 0.624603] pnp: PnP ACPI: found 6 devices [ 0.638626] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.641293] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.642988] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.644583] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.646697] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.648112] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.650189] NET: Registered protocol family 2 [ 0.652187] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.655922] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.659693] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.663879] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.666960] TCP: Hash tables configured (established 65536 bind 65536) [ 0.669654] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.671835] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.674116] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.676249] NET: Registered protocol family 1 [ 0.678892] RPC: Registered named UNIX socket transport module. [ 0.681211] RPC: Registered udp transport module. [ 0.682592] RPC: Registered tcp transport module. [ 0.683613] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.685063] NET: Registered protocol family 44 [ 0.686113] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.687656] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.689491] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.691499] PCI: CLS 0 bytes, default 64 [ 0.692745] Unpacking initramfs... [ 2.231833] debug: unmapping init [mem 0xffff97b73cc64000-0xffff97b73ffcffff] [ 2.235899] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.237671] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.240721] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.941153] Initialise system trusted keyrings [ 2.944792] Key type blacklist registered [ 2.948398] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.963828] zbud: loaded [ 2.967896] *** VALIDATE nfs *** [ 2.969123] *** VALIDATE nfs4 *** [ 2.971384] pstore: using deflate compression [ 2.975120] Platform Keyring initialized [ 3.105882] NET: Registered protocol family 38 [ 3.108557] Key type asymmetric registered [ 3.114053] Asymmetric key parser 'x509' registered [ 3.116466] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.129538] io scheduler mq-deadline registered [ 3.132103] io scheduler kyber registered [ 3.135368] io scheduler bfq registered [ 3.140491] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.145941] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.149514] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.152885] ACPI: Power Button [PWRF] [ 3.160393] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.171279] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.184912] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.216291] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.246770] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.253152] Non-volatile memory driver v1.3 [ 3.255213] Linux agpgart interface v0.103 [ 3.305529] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.311922] vda: detected capacity change from 0 to 76779520 [ 3.339619] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.345985] vdb: detected capacity change from 0 to 1073741824 [ 3.355277] libphy: Fixed MDIO Bus: probed [ 3.359654] usbcore: registered new interface driver usbserial_generic [ 3.362148] usbserial: USB Serial support registered for generic [ 3.364352] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.370463] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.374117] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.379334] mousedev: PS/2 mouse device common for all mice [ 3.382631] rtc_cmos 00:05: RTC can wake from S4 [ 3.388884] rtc_cmos 00:05: registered as rtc0 [ 3.391518] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.396929] intel_pstate: CPU model not supported [ 3.400621] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.406108] hid: raw HID events driver (C) Jiri Kosina [ 3.407529] usbcore: registered new interface driver usbhid [ 3.409643] usbhid: USB HID core driver [ 3.411398] drop_monitor: Initializing network drop monitor service [ 3.417895] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.418965] Initializing XFRM netlink socket [ 3.430206] NET: Registered protocol family 10 [ 3.433490] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.433615] Segment Routing with IPv6 [ 3.443881] NET: Registered protocol family 17 [ 3.446287] mpls_gso: MPLS GSO support [ 3.452368] RAS: Correctable Errors collector initialized. [ 3.455205] AVX version of gcm_enc/dec engaged. [ 3.456892] AES CTR mode by8 optimization enabled [ 3.540552] sched_clock: Marking stable (3540477459, 0)->(4274295255, -733817796) [ 3.546220] registered taskstats version 1 [ 3.548334] Loading compiled-in X.509 certificates [ 3.551263] zswap: loaded using pool lzo/zbud [ 3.580815] Key type big_key registered [ 3.594820] Key type encrypted registered [ 3.596696] ima: No TPM chip found, activating TPM-bypass! [ 3.599466] ima: Allocated hash algorithm: sha1 [ 3.601445] ima: No architecture policies found [ 3.604271] evm: Initialising EVM extended attributes: [ 3.606567] evm: security.selinux [ 3.607965] evm: security.ima [ 3.611522] evm: security.capability [ 3.613527] evm: HMAC attrs: 0x1 [ 3.616695] rtc_cmos 00:05: setting system clock to 2026-09-13 16:26:02 UTC (1789316762) [ 3.626871] debug: unmapping init [mem 0xffffffffa3003000-0xffffffffa31fffff] [ 3.630865] debug: unmapping init [mem 0xffffffffa1d82000-0xffffffffa2058fff] [ 3.642083] Write protecting the kernel read-only data: 28672k [ 3.648505] debug: unmapping init [mem 0xffffffffa0403000-0xffffffffa05fffff] [ 3.656792] debug: unmapping init [mem 0xffffffffa0d14000-0xffffffffa0dfffff] [ 3.693922] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.710447] systemd[1]: Detected virtualization kvm. [ 3.712799] systemd[1]: Detected architecture x86-64. [ 3.716191] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.746281] systemd[1]: No hostname configured. [ 3.747695] systemd[1]: Set hostname to . [ 3.754224] random: systemd: uninitialized urandom read (16 bytes read) [ 3.755807] systemd[1]: Initializing machine ID from random generator. [ 3.805825] random: ln: uninitialized urandom read (6 bytes read) [ 3.898575] random: systemd: uninitialized urandom read (16 bytes read) [ 3.901032] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.908501] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.915536] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.721456] device-mapper: uevent: version 1.0.3 [ 4.726187] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.846513] scsi host0: ata_piix [ 5.945488] scsi host1: ata_piix [ 5.980322] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.983128] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 5.987908] virtio_net virtio0 ens2: renamed from eth0 [ 9.334730] random: fast init done [ 10.305141] random: crng init done [ 10.307676] random: 7 urandom warning(s) missed due to ratelimiting [ 12.489964] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.838628] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.615477] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.289918] SELinux: Disabled at runtime. [ 16.425176] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.452641] systemd[1]: Detected virtualization kvm. [ 16.458063] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.339459] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.343578] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.353798] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.358168] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.363253] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.370766] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.384484] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ 17.858761] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ 18.513211] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.630797] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.680708] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.184248] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.226445] EDAC sbridge: Ver: 1.1.2 [ 22.689878] Key type dns_resolver registered [ 23.071436] NFS: Registering the id_resolver key type [ 23.074216] Key type id_resolver registered [ 23.076879] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg337-client login: [ 66.323658] libcfs: loading out-of-tree module taints kernel. [ 66.415141] Key type ._llcrypt registered [ 66.418032] Key type .llcrypt registered [ 66.691556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 66.699240] alg: No test for adler32 (adler32-zlib) [ 67.733423] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 68.054907] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 69.679707] Key type lgssc registered [ 70.354336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.131176] Lustre: Mounted lustre-client - version 2.17.58_39_gbf2c64c [ 124.572558] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 132.374260] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing check_logdir /tmp/testlogs/ [ 133.958140] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing yml_node [ 135.715081] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 136.655976] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 137.646329] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 138.278657] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Sep 13 12:28:16 EDT 2026 [ 143.591675] Lustre: Unmounted lustre-client [ 155.803261] Key type lgssc unregistered [ 155.933500] LNet: 8401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 155.937539] LNetError: 8401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 155.946346] LNet: Removed LNI 192.168.203.37@tcp [ 156.247197] Key type .llcrypt unregistered [ 156.248675] Key type ._llcrypt unregistered [ 166.213303] Key type ._llcrypt registered [ 166.215263] Key type .llcrypt registered [ 166.294502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 166.302525] alg: No test for adler32 (adler32-zlib) [ 167.123874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 167.158272] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 167.168461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 168.788380] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 168.868351] LNet: 9277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 168.872426] LNetError: 9277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 168.881521] LNet: Removed LNI 192.168.203.37@tcp [ 169.165495] Key type .llcrypt unregistered [ 169.167816] Key type ._llcrypt unregistered [ 171.485831] Key type ._llcrypt registered [ 171.487566] Key type .llcrypt registered [ 172.062983] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:28:50 (1789316930) [ 172.180768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 172.188167] alg: No test for adler32 (adler32-zlib) [ 173.008396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 173.049262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 173.060142] LNet: 9684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 173.070748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 173.096909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 173.107378] LNet: 9691:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 175.921186] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:28:54 (1789316934) [ 175.958946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 175.971616] LNet: 10261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 175.986573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 176.025336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 178.787822] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:28:57 (1789316937) [ 178.831892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.843417] LNet: 10848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 178.854938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 178.892275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 181.484157] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:28:59 (1789316939) [ 181.542829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.554875] LNet: 11435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 181.567391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.602978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 184.089059] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:29:02 (1789316942) [ 184.132633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.154391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.187467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 184.349906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 186.629573] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:29:05 (1789316945) [ 186.657953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 186.668202] LNet: 12602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 186.671715] LNet: 12602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 186.681362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.716664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 186.870770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 187.036730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 189.414679] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:29:07 (1789316947) [ 189.457426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.479501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.518635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 191.882249] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:29:10 (1789316950) [ 191.947403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.965660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.991558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 192.007427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 192.150373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 192.164530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 192.321726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 192.337478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 192.492648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 192.509701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 192.666984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 192.691144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 192.848929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 192.863708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 193.010214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 193.026810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 193.181608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 193.196378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 193.348427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 193.364382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 193.512585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 193.530552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 193.678281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 193.693482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 193.849689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 193.867739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 194.019906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 194.035828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 196.363206] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:29:14 (1789316954) [ 196.418805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.432752] LNet: 14467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 196.436680] LNet: 14467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 196.448821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.481450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 196.496694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 198.928075] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:29:17 (1789316957) [ 198.960806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.979547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.014737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 199.029875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 201.324082] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:29:19 (1789316959) [ 201.353862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 201.371727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 201.409048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 201.425575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 203.898345] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:29:22 (1789316962) [ 203.953735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 203.974260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.007856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 204.030438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 206.485146] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:29:24 (1789316964) [ 206.538943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.557560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.589156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 206.607427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 209.193369] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:29:27 (1789316967) [ 209.245231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.263788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.299444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 209.317621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 211.865027] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:29:30 (1789316970) [ 211.921696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.940466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.979917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 212.156894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 214.752196] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:29:33 (1789316973) [ 214.817810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.830380] LNet: 18575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 214.832564] LNet: 18575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 214.844095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.936988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 215.032786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 217.757527] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:29:36 (1789316976) [ 217.802337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.826040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.892987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 217.989049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 220.526778] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:29:38 (1789316978) [ 220.591395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.613882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.812706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 220.921664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 223.500996] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:29:41 (1789316981) [ 223.563112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.579682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.614836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 223.637779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 226.376959] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:29:44 (1789316984) [ 226.431599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.450528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.485499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 226.513201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 229.293206] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:29:47 (1789316987) [ 229.368099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.390697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.423741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 232.364592] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:29:50 (1789316990) [ 232.430511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.451083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.484413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 235.081136] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:29:53 (1789316993) [ 235.127828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.153953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.190921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 235.358048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 237.911378] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:29:56 (1789316996) [ 237.962387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 237.989621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.026405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 238.048102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 240.621092] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:29:58 (1789316998) [ 240.652374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.673647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.712609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 240.730094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 243.413286] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:30:01 (1789317001) [ 243.463848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.483842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.522642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 243.543648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 246.254868] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:30:04 (1789317004) [ 246.332330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.351232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.387446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 246.406465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 249.057138] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:30:07 (1789317007) [ 249.108528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.117943] LNet: 25756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 249.121562] LNet: 25756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 249.135581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.160804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 249.189820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 249.200112] LNetError: 25770:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 249.219838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 249.238901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 251.572233] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:30:09 (1789317009) [ 251.642261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 251.659051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 251.690982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.HmhmU7Z3rt [ 251.723227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.HmhmU7Z3rt [ 251.746551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.HmhmU7Z3rt [ 254.142649] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:30:12 (1789317012) [ 254.181475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.201035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.223304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 254.238322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 254.259142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 256.718571] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:30:15 (1789317015) [ 256.766376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 256.788805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 256.828296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 256.840940] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 257.444964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 257.929979] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 259.623207] Key type lgssc registered [ 260.032890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 261.437115] LNetError: 27529:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.37@tcp99 has no available nets [ 262.585732] Key type lgssc unregistered [ 262.741201] LNetError: 28635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 262.753935] LNet: Removed LNI 192.168.203.37@tcp [ 263.012590] Key type .llcrypt unregistered [ 263.014359] Key type ._llcrypt unregistered [ 265.281577] Key type ._llcrypt registered [ 265.283215] Key type .llcrypt registered [ 265.785511] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:30:24 (1789317024) [ 265.983315] Key type .llcrypt unregistered [ 265.985144] Key type ._llcrypt unregistered [ 268.385484] Key type ._llcrypt registered [ 268.386890] Key type .llcrypt registered [ 268.458183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 268.465229] alg: No test for adler32 (adler32-zlib) [ 269.274278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.310899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 269.322300] LNet: 29543:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 269.326433] LNet: 29543:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 269.330182] LNet: 29543:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 269.337173] LNet: Message forwarding will be enabled in 2s [ 272.415135] LNet: Message forwarding enabled [ 272.512958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 272.523222] LNet: Message forwarding will stop in 2s to 6s [ 276.514905] LNet: Message forwarding disabled [ 276.715815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 276.725767] LNet: 29641:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 276.729206] LNet: 29641:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 276.733541] LNet: 29641:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 276.741779] LNet: Message forwarding will be enabled in 2s [ 276.806494] LNet: 29654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 277.088430] Key type .llcrypt unregistered [ 277.090840] Key type ._llcrypt unregistered [ 281.306412] Key type ._llcrypt registered [ 281.308591] Key type .llcrypt registered [ 281.862230] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:30:40 (1789317040) [ 282.018708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 282.026690] alg: No test for adler32 (adler32-zlib) [ 282.840047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.880549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 285.234180] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:30:43 (1789317043) [ 285.281610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.292735] LNet: 31050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.304483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.331326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 285.346539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 285.363726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 285.379325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 285.394900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 285.411584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 285.427644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 285.442598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 285.458329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 285.473628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 285.490936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 285.504768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 285.521626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 285.537511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 285.553392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 285.569902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 285.587900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 285.605596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 285.622561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 285.632826] LNetError: 31095:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 288.003114] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:30:46 (1789317046) [ 288.035307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.045993] LNet: 31659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.059371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.100088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 290.534260] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:30:48 (1789317048) [ 291.268224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg337-server [ 291.939108] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:30:50 (1789317050) [ 292.002268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.015931] LNet: 32595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.028062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.053473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 292.066643] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 292.426464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[38-42]@tcp [ 292.441213] LNet: 32728:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 292.460789] LNetError: 32599:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.38@tcp is being used as a gateway but routing feature is not turned on [ 292.471439] LNetError: 32599:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.38@tcp has gone from up to down [ 295.064478] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:30:53 (1789317053) [ 295.519424] LNetError: 32606:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.40@tcp has gone from up to down [ 295.529345] LNetError: 32606:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 295.808275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg337-server [ 296.504051] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:30:54 (1789317054) [ 296.564589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.575785] LNet: 33662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 296.580900] LNetError: 33662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 296.592585] LNet: Removed LNI 192.168.203.37@tcp [ 296.603454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.629029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 296.640339] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 296.919062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[38-43/2]@tcp [ 296.931557] LNet: 33779:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 296.935462] LNet: 33779:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 296.942555] LNetError: 33666:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.38@tcp is being used as a gateway but routing feature is not turned on [ 296.945513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[38-43/2]@tcp [ 296.951240] LNetError: 33666:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 296.959914] LNetError: 33666:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.38@tcp has gone from up to down [ 296.964645] LNetError: 33666:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 299.597609] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:30:57 (1789317057) [ 299.660117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.671935] LNetError: 34370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.686612] LNet: Removed LNI 192.168.203.37@tcp [ 299.696603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.737478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 299.760692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.777852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.812848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 299.837598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.855339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.881203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 299.917082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.938635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.971445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.006139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.029601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.056114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 300.091408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.110714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.142628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.176557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.196407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.228215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 300.263756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.288260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.320504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.352754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.371253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.398235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 300.435713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.457211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.491580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.523865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.542731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.568637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 300.592395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.602349] LNet: 34516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.605736] LNet: 34516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 300.616614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.648083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.525254] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:31:01 (1789317061) [ 304.466267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg337-server [ 305.271778] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:31:03 (1789317063) [ 306.244740] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg337-server [ 307.081777] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:31:05 (1789317065) [ 307.587492] Key type .llcrypt unregistered [ 307.590621] Key type ._llcrypt unregistered [ 310.395919] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 310.487783] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 310.490931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 310.492740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 312.706312] Key type ._llcrypt registered [ 312.707555] Key type .llcrypt registered [ 312.781458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 312.788233] alg: No test for adler32 (adler32-zlib) [ 313.606416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.634260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 313.646377] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 314.003792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 314.016677] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 315.053991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 315.065322] LNet: Removed LNI 10.1.2.3@tcp [ 315.119866] LNet: 36486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.126743] LNetError: 36486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 315.137557] LNet: Removed LNI 192.168.203.37@tcp [ 315.436538] Key type .llcrypt unregistered [ 315.438093] Key type ._llcrypt unregistered [ 321.210622] Key type ._llcrypt registered [ 321.212297] Key type .llcrypt registered [ 321.840997] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:31:20 (1789317080) [ 322.081641] Key type .llcrypt unregistered [ 322.083920] Key type ._llcrypt unregistered [ 324.857641] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 324.956501] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 324.960508] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 324.963469] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 327.193367] Key type ._llcrypt registered [ 327.194656] Key type .llcrypt registered [ 327.273749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 327.282801] alg: No test for adler32 (adler32-zlib) [ 328.102410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.128453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 328.139912] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 328.459619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 328.472074] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 329.533880] LNet: 37972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.537475] LNetError: 37972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 329.541766] LNet: Removed LNI 10.1.2.3@tcp [ 329.554329] LNet: Removed LNI 192.168.203.37@tcp [ 329.833727] Key type .llcrypt unregistered [ 329.835804] Key type ._llcrypt unregistered [ 334.526255] Key type ._llcrypt registered [ 334.527538] Key type .llcrypt registered [ 335.174466] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:31:33 (1789317093) [ 335.427495] Key type .llcrypt unregistered [ 335.432356] Key type ._llcrypt unregistered [ 338.094128] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 338.205824] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 338.209478] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 338.212401] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 340.467095] Key type ._llcrypt registered [ 340.468479] Key type .llcrypt registered [ 340.566924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 340.575472] alg: No test for adler32 (adler32-zlib) [ 341.425675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.460769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 341.477907] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 342.433363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 342.444250] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 343.309548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 343.320696] LNet: Removed LNI 10.1.2.3@tcp [ 343.329042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 343.350649] LNet: Removed LNI 10.1.2.31@tcp [ 343.362746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 343.379824] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 343.714482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 343.728878] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 344.179106] LNet: 39886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.182247] LNetError: 39886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 344.185256] LNet: Removed LNI 10.1.2.31@tcp [ 344.190597] LNet: Removed LNI 10.1.2.3@tcp [ 344.475022] Key type .llcrypt unregistered [ 344.477556] Key type ._llcrypt unregistered [ 349.115171] Key type ._llcrypt registered [ 349.117205] Key type .llcrypt registered [ 349.684674] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:31:47 (1789317107) [ 349.854460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 349.862341] alg: No test for adler32 (adler32-zlib) [ 350.682217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.711714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 350.727216] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 351.088699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.101792] LNet: 40843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.105738] LNetError: 40843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 351.120972] LNet: Removed LNI 192.168.203.37@tcp [ 351.133545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.163403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.37@tcp [ 351.193414] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 351.567401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.37@tcp [ 351.605597] LNet: Removed LNI 192.168.203.37@tcp [ 351.635114] LNet: 40978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.905441] Key type .llcrypt unregistered [ 351.907051] Key type ._llcrypt unregistered [ 356.160939] Key type ._llcrypt registered [ 356.162505] Key type .llcrypt registered [ 356.711982] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:31:54 (1789317114) [ 357.462526] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg337-server [ 358.164190] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:31:56 (1789317116) [ 358.387612] Key type .llcrypt unregistered [ 358.389656] Key type ._llcrypt unregistered [ 360.702600] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 360.766149] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 360.770217] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 360.775817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 362.980818] Key type ._llcrypt registered [ 362.982066] Key type .llcrypt registered [ 363.066339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 363.072623] alg: No test for adler32 (adler32-zlib) [ 363.893217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.922172] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 364.264854] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 365.204368] LNet: 42806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.207889] LNetError: 42806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 365.212536] LNet: Removed LNI 192.168.203.37@tcp1 [ 365.218713] LNet: Removed LNI 10.1.2.3@tcp2 [ 365.473544] Key type .llcrypt unregistered [ 365.475932] Key type ._llcrypt unregistered [ 369.830165] Key type ._llcrypt registered [ 369.831481] Key type .llcrypt registered [ 370.377957] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:32:08 (1789317128) [ 370.538275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 370.545570] alg: No test for adler32 (adler32-zlib) [ 371.368420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.396781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 371.411257] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 371.805276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.817521] LNet: 43772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.821465] LNetError: 43772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 371.834592] LNet: Removed LNI 192.168.203.37@tcp [ 371.843392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.867924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 371.881569] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 374.660500] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:32:12 (1789317132) [ 374.734172] LNet: 44361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.740712] LNetError: 44361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 374.753573] LNet: Removed LNI 192.168.203.37@tcp [ 375.005550] Key type .llcrypt unregistered [ 375.007351] Key type ._llcrypt unregistered [ 377.174669] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 377.236154] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 377.240586] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 377.243207] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 379.456987] Key type ._llcrypt registered [ 379.458360] Key type .llcrypt registered [ 379.533570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 379.539620] alg: No test for adler32 (adler32-zlib) [ 380.359632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.387327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 380.402665] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 380.711579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 380.726546] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 381.653481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.664644] LNet: 45095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.667876] LNetError: 45095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 381.671200] LNet: Removed LNI 10.1.2.3@tcp [ 381.676530] LNet: Removed LNI 192.168.203.37@tcp [ 381.686233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.713595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 381.724556] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 381.728175] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 381.912273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.922263] LNet: Removed LNI 10.1.2.3@tcp [ 381.928416] LNet: Removed LNI 192.168.203.37@tcp [ 381.937736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.965688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 381.980660] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 381.983739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 382.155183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 382.165851] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 382.595519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.605261] LNet: 45285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.608053] LNet: 45285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 382.611774] LNetError: 45285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 382.614576] LNetError: 45285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 382.617256] LNet: Removed LNI 10.1.2.3@tcp [ 382.618499] LNet: Removed LNI 192.168.203.37@tcp [ 382.625695] LNet: Removed LNI 192.168.203.37@tcp2 [ 382.639244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.674053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 382.687117] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 382.689575] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 382.692168] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 385.275533] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:32:23 (1789317143) [ 385.307800] LNet: 45873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.311209] LNetError: 45873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 385.314269] LNetError: 45873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 385.317270] LNet: Removed LNI 10.1.2.3@tcp [ 385.318795] LNet: Removed LNI 192.168.203.37@tcp [ 385.327731] LNet: Removed LNI 192.168.203.37@tcp2 [ 385.591704] Key type .llcrypt unregistered [ 385.593276] Key type ._llcrypt unregistered [ 387.877454] Key type ._llcrypt registered [ 387.879127] Key type .llcrypt registered [ 387.957482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 387.964930] alg: No test for adler32 (adler32-zlib) [ 388.783124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.813379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 388.833112] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 389.224277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.233426] LNet: 46272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.236391] LNetError: 46272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.247076] LNet: Removed LNI 192.168.203.37@tcp [ 389.257509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.287519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 389.302619] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 389.498837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.513939] LNet: Removed LNI 192.168.203.37@tcp [ 389.527271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.562912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 389.575565] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 389.771505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.782410] LNet: 46332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.785330] LNet: 46332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 389.789042] LNetError: 46332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.792989] LNetError: 46332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 389.803514] LNet: Removed LNI 192.168.203.37@tcp [ 389.813831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.845263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 389.859770] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 390.061276] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 390.158123] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 390.160843] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 390.163808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 392.297280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.307515] LNet: 46410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.311926] LNetError: 46410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 392.321796] LNet: Removed LNI 192.168.203.37@tcp [ 392.332570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.360507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 392.373452] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 395.005753] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:32:33 (1789317153) [ 395.060879] LNet: 47001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.065770] LNetError: 47001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 395.074503] LNet: Removed LNI 192.168.203.37@tcp [ 395.345792] Key type .llcrypt unregistered [ 395.347745] Key type ._llcrypt unregistered [ 397.626556] Key type ._llcrypt registered [ 397.627977] Key type .llcrypt registered [ 397.693041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 397.700774] alg: No test for adler32 (adler32-zlib) [ 398.519780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.546345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 398.558156] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 398.889952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.903736] LNet: 47401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.907164] LNetError: 47401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 398.917411] LNet: Removed LNI 192.168.203.37@tcp [ 398.928066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.956799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 398.967594] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 401.307160] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:32:39 (1789317159) [ 401.350333] LNet: 47987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.354119] LNetError: 47987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 401.362620] LNet: Removed LNI 192.168.203.37@tcp [ 401.604711] Key type .llcrypt unregistered [ 401.608750] Key type ._llcrypt unregistered [ 403.737205] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 403.829120] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 403.831626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 403.834274] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 406.017351] Key type ._llcrypt registered [ 406.018982] Key type .llcrypt registered [ 406.102677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 406.109092] alg: No test for adler32 (adler32-zlib) [ 406.921341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.944346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 406.956758] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 407.296704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.307320] LNet: 48438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.310548] LNetError: 48438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 407.322516] LNet: Removed LNI 192.168.203.37@tcp [ 407.331648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.360248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 407.372272] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 409.839666] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:32:48 (1789317168) [ 409.886313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.896917] LNet: 49035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.900888] LNetError: 49035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 409.910440] LNet: Removed LNI 192.168.203.37@tcp [ 409.921260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.961193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 412.264469] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:32:50 (1789317170) [ 412.301691] LNet: 49612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.552155] Key type .llcrypt unregistered [ 412.553738] Key type ._llcrypt unregistered [ 414.900837] Key type ._llcrypt registered [ 414.902267] Key type .llcrypt registered [ 414.972832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 414.980484] alg: No test for adler32 (adler32-zlib) [ 415.792409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.815970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 415.827658] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 416.165098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.175439] LNet: 50012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.178187] LNetError: 50012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 416.196430] LNet: Removed LNI 192.168.203.37@tcp [ 416.204876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.232249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 416.243931] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 418.637898] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:32:57 (1789317177) [ 418.669970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.681323] LNet: 50598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.684812] LNetError: 50598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 418.696341] LNet: Removed LNI 192.168.203.37@tcp [ 418.707678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.741566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 420.815455] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:32:59 (1789317179) [ 421.319640] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 421.884363] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:33:00 (1789317180) [ 422.367195] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 422.960769] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:33:01 (1789317181) [ 423.481648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 424.064624] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:33:02 (1789317182) [ 424.609794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 425.222871] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:33:03 (1789317183) [ 425.774373] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 426.406560] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:33:04 (1789317184) [ 426.454905] LNet: 52618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.718350] Key type .llcrypt unregistered [ 426.720181] Key type ._llcrypt unregistered [ 428.944370] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 429.011905] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 429.014644] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 429.017669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 431.226087] Key type ._llcrypt registered [ 431.227575] Key type .llcrypt registered [ 431.303577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 431.312159] alg: No test for adler32 (adler32-zlib) [ 432.121530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.157139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 432.168490] LNet: Added LNI 192.168.203.37@tcp [8/512/0/180] [ 432.171733] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 432.173994] LNet: 52941:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 432.213854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.223308] LNet: 52960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.226616] LNetError: 52960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 432.229794] LNet: Removed LNI 10.1.2.3@tcp [ 432.231131] LNetError: 52960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.242127] LNet: Removed LNI 192.168.203.37@tcp [ 432.250732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.278499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 432.292393] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 432.320730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.331251] LNet: 52984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.334626] LNetError: 52984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.347460] LNet: Removed LNI 192.168.203.37@tcp [ 432.356785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.384788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 432.395037] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 432.397398] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 432.399109] LNet: 52992:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 432.436790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.445686] LNet: 53010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.448728] LNetError: 53010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 432.453336] LNet: Removed LNI 10.1.2.3@tcp [ 432.455165] LNetError: 53010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.465867] LNet: Removed LNI 192.168.203.37@tcp [ 432.475758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.504809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 432.516715] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 432.519633] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 432.521621] LNet: 53018:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 432.587142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.595480] LNet: 53044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.598588] LNetError: 53044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 432.601856] LNet: Removed LNI 10.1.2.3@tcp [ 432.603690] LNetError: 53044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.614319] LNet: Removed LNI 192.168.203.37@tcp [ 432.623570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.651220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 432.662892] LNet: Added LNI 192.168.203.37@tcp [8/256/0/182] [ 432.665709] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 432.706433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.717989] LNet: 53072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.722250] LNetError: 53072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 432.725604] LNet: Removed LNI 10.1.2.3@tcp [ 432.727337] LNetError: 53072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.734456] LNet: Removed LNI 192.168.203.37@tcp [ 432.742939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.770397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 432.782290] LNet: Added LNI 192.168.203.37@tcp [8/256/0/182] [ 432.784679] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 432.786640] LNet: 53080:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 432.818469] LNet: Removed LNI 10.1.2.3@tcp [ 432.831520] LNet: Removed LNI 192.168.203.37@tcp [ 433.065680] Key type .llcrypt unregistered [ 433.067120] Key type ._llcrypt unregistered [ 436.967391] Key type ._llcrypt registered [ 436.968725] Key type .llcrypt registered [ 437.477580] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:33:15 (1789317195) [ 438.018131] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 438.636303] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:33:16 (1789317196) [ 438.774717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 438.781123] alg: No test for adler32 (adler32-zlib) [ 439.597773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.625780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 439.638508] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 439.953649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.37 [ 439.964121] LNet: 54339:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 439.968623] LNet: Removed LNI 192.168.203.37@tcp [ 442.339230] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:33:20 (1789317200) [ 442.392104] LNet: 54905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.395218] LNetError: 54905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 442.402425] LNet: Removed LNI 192.168.203.37@tcp [ 442.655397] Key type .llcrypt unregistered [ 442.656811] Key type ._llcrypt unregistered [ 444.859587] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 444.986871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 447.296287] Key type ._llcrypt registered [ 447.297964] Key type .llcrypt registered [ 447.357405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 447.363637] alg: No test for adler32 (adler32-zlib) [ 448.329284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 450.607217] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:33:28 (1789317208) [ 450.651889] LNet: 55813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.655522] LNetError: 55813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 450.666661] LNet: Removed LNI 10.1.2.3@tcp [ 450.932509] Key type .llcrypt unregistered [ 450.934295] Key type ._llcrypt unregistered [ 453.090870] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 453.239169] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 453.241251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 453.243897] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 455.534251] Key type ._llcrypt registered [ 455.536014] Key type .llcrypt registered [ 455.606923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 455.613728] alg: No test for adler32 (adler32-zlib) [ 456.438383] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 458.749846] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:33:37 (1789317217) [ 459.257242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 459.782918] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:33:38 (1789317218) [ 459.814738] LNet: 56989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.817882] LNetError: 56989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 459.826463] LNet: Removed LNI 10.1.2.3@tcp [ 460.039436] Key type .llcrypt unregistered [ 460.040805] Key type ._llcrypt unregistered [ 462.144563] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 462.275740] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 462.278716] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 462.282586] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 464.580975] Key type ._llcrypt registered [ 464.582493] Key type .llcrypt registered [ 464.655152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 464.662236] alg: No test for adler32 (adler32-zlib) [ 465.483277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.518386] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 468.677902] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:33:47 (1789317227) [ 468.846087] LNet: 58220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.850108] LNetError: 58220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 468.866584] LNet: Removed LNI 10.1.2.3@tcp [ 469.108373] Key type .llcrypt unregistered [ 469.110094] Key type ._llcrypt unregistered [ 471.295069] Key type ._llcrypt registered [ 471.296582] Key type .llcrypt registered [ 471.374462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 471.382632] alg: No test for adler32 (adler32-zlib) [ 472.190013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 472.204232] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 475.409230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 475.668397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_interrupt -r 1 [ 475.687854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_interrupt -r 1 [ 475.708890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 475.720330] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 475.805082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 475.819109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 475.835255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 475.849895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 475.864844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 475.877481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 475.895973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 475.913170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 475.928549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 475.939161] LNet: 58655:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 475.950203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 475.968907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 475.982960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 475.997806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 476.016425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 476.035696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 476.053507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 476.270368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_dropped -r 1 [ 476.289207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_dropped -r 1 [ 476.305900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 476.315104] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 476.319510] LNet: 1 local NIs in recovery (showing 1): 192.168.203.37@tcp [ 476.403796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 476.417750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 476.433842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 476.448683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 476.464835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 476.480877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 476.496306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 476.511949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 476.527792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 476.537746] LNet: 58767:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 476.547798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 476.563405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 476.577349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 476.591504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 476.607210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 476.623711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 476.641499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 476.859838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_aborted -r 1 [ 476.880115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_aborted -r 1 [ 476.896616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 476.993929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 477.010317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 477.025703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 477.041064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 477.055199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 477.070112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 477.086614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 477.103368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 477.118968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 477.133916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 477.147542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 477.163209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 477.177646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 477.193650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 477.210382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 477.226552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 477.442586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_no_route -r 1 [ 477.460315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_no_route -r 1 [ 477.485159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 477.495538] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 477.502042] LNet: Skipped 2 previous similar messages [ 477.593227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 477.609049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 477.625920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 477.640641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 477.655852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 477.672880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 477.689281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 477.705941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 477.724068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 477.736641] LNet: 58989:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 477.740681] LNet: 58989:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 477.749333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 477.764325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 477.778952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 477.793369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 477.808530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 477.823625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 477.840078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 478.027668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_timeout -r 1 [ 478.046751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_timeout -r 1 [ 478.066542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 478.176254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 478.190475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 478.204629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 478.219787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 478.236600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 478.253408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 478.268824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 478.286184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 478.300907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 478.316918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 478.337528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 478.351958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 478.367823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 478.386780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 478.402564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 478.422451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 478.640743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_error -r 1 [ 478.659714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_error -r 1 [ 478.676885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 478.774101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 478.788109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 478.802420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 478.817481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 478.832944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 478.847406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 478.863641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 478.882665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 478.899197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 478.915701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 478.930065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 478.947781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 478.961554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 478.977077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 478.992233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 479.008527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 479.120190] LNet: 59252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 479.123712] LNetError: 59252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 479.132460] LNet: Removed LNI 192.168.203.37@tcp [ 479.293607] Key type .llcrypt unregistered [ 479.295516] Key type ._llcrypt unregistered [ 485.360880] Key type ._llcrypt registered [ 485.362238] Key type .llcrypt registered [ 485.857809] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:34:04 (1789317244) [ 486.144566] Key type .llcrypt unregistered [ 486.146034] Key type ._llcrypt unregistered [ 488.235978] Key type ._llcrypt registered [ 488.237927] Key type .llcrypt registered [ 488.313585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 488.320260] alg: No test for adler32 (adler32-zlib) [ 489.129602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 489.141413] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 492.154470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 495.593721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.609633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 495.618993] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 496.125527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_interrupt -r 1 [ 496.141865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_interrupt -r 1 [ 496.157929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_interrupt -r 1 [ 496.174673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_interrupt -r 1 [ 496.192268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_interrupt -r 1 [ 496.210575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_interrupt -r 1 [ 496.228711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_interrupt -r 1 [ 496.250663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_interrupt -r 1 [ 496.270808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 496.281329] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 496.284913] LNet: 1 local NIs in recovery (showing 1): 192.168.203.37@tcp [ 496.368877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 496.383294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 496.396178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 496.409552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 496.423607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 496.439053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 496.456448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 496.470583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 496.484940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 496.496564] LNet: 60712:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 496.504841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 496.519128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 496.534381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 496.549505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 496.565480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 496.580280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 496.593575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 496.775043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_dropped -r 1 [ 496.791143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_dropped -r 1 [ 496.808419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_dropped -r 1 [ 496.826234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_dropped -r 1 [ 496.842808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_dropped -r 1 [ 496.859322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_dropped -r 1 [ 496.875775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_dropped -r 1 [ 496.892658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_dropped -r 1 [ 496.910122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 496.920423] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 496.923641] LNet: Skipped 2 previous similar messages [ 497.004759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 497.018619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 497.035060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 497.050882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 497.066626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 497.080239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 497.093953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 497.106370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 497.120670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 497.128468] LNet: 60835:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 497.137243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 497.152707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 497.165942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 497.178953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 497.193754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 497.207595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 497.222089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 497.425216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_aborted -r 1 [ 497.444736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_aborted -r 1 [ 497.463387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_aborted -r 1 [ 497.480238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_aborted -r 1 [ 497.496557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_aborted -r 1 [ 497.512395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_aborted -r 1 [ 497.528250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_aborted -r 1 [ 497.544364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_aborted -r 1 [ 497.562097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 497.647932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 497.661368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 497.675867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 497.690324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 497.705730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 497.720344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 497.734418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 497.746355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 497.760299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 497.774836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 497.788917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 497.803309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 497.816564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 497.831072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 497.845213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 497.858903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 498.045036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_no_route -r 1 [ 498.062893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_no_route -r 1 [ 498.079706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_no_route -r 1 [ 498.098254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_no_route -r 1 [ 498.118272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_no_route -r 1 [ 498.135485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_no_route -r 1 [ 498.151383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_no_route -r 1 [ 498.167697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_no_route -r 1 [ 498.185075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 498.194527] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 498.198054] LNet: Skipped 6 previous similar messages [ 498.279830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 498.293396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 498.309161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 498.323731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 498.337815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 498.351335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 498.364445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 498.378598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 498.391138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 498.399712] LNet: 61081:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 498.403391] LNet: 61081:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 498.412128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 498.424220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 498.438631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 498.452128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 498.466473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 498.481215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 498.498204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 498.709532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_timeout -r 1 [ 498.724358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_timeout -r 1 [ 498.741634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_timeout -r 1 [ 498.758227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_timeout -r 1 [ 498.773162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_timeout -r 1 [ 498.787229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_timeout -r 1 [ 498.803793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_timeout -r 1 [ 498.820632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_timeout -r 1 [ 498.838209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 498.922971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 498.936710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 498.950439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 498.963233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 498.977032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 498.991572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 499.006928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 499.021310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 499.034742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 499.048192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 499.062468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 499.075917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 499.088682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 499.103256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 499.116869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 499.131894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 499.310049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_error -r 1 [ 499.327661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_error -r 1 [ 499.344157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_error -r 1 [ 499.359509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_error -r 1 [ 499.375359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_error -r 1 [ 499.392252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_error -r 1 [ 499.408615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_error -r 1 [ 499.424385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_error -r 1 [ 499.441043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 499.526548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 499.540540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 499.554904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 499.569910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 499.584693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 499.599971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 499.615272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 499.630093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 499.646199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 499.659795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 499.674775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 499.688194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 499.704464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 499.721281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 499.734789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 499.750658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 501.031113] LNet: 61402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.033746] LNetError: 61402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 501.037400] LNet: Removed LNI 192.168.203.37@tcp [ 502.053420] LNet: Removed LNI 192.168.203.37@tcp1 [ 502.205612] Key type .llcrypt unregistered [ 502.207230] Key type ._llcrypt unregistered [ 507.921020] Key type ._llcrypt registered [ 507.922983] Key type .llcrypt registered [ 508.400223] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:34:26 (1789317266) [ 508.703349] Key type .llcrypt unregistered [ 508.705117] Key type ._llcrypt unregistered [ 510.844149] Key type ._llcrypt registered [ 510.845618] Key type .llcrypt registered [ 510.915663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 510.921732] alg: No test for adler32 (adler32-zlib) [ 511.736187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 511.750444] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 514.905576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 515.164728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e remote_dropped -r 1 [ 515.183397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e remote_dropped -r 1 [ 515.199291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 515.209304] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 515.290388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 515.306301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 515.322287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 515.336928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 515.352600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 515.366744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 515.384727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 515.403354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 515.420610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 515.430942] LNet: 62645:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 515.442945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 515.459202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 515.474148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 515.488910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 515.505128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 515.523990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 515.541810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 515.755415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e remote_error -r 1 [ 515.772992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e remote_error -r 1 [ 515.790488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 515.801374] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 515.891233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 515.908461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 515.923925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 515.940873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 515.955456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 515.970465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 515.985369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 516.001556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 516.016578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 516.026670] LNet: 62756:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 516.035684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 516.050439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 516.065733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 516.081307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 516.098387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 516.114137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 516.129841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 516.334213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e remote_timeout -r 1 [ 516.349471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e remote_timeout -r 1 [ 516.365968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 516.455549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 516.470432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 516.486347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 516.502152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 516.517589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 516.532555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 516.547683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 516.561984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 516.578947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 516.595217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 516.611140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 516.627471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 516.642949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 516.658248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 516.673284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 516.689936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 516.799484] LNet: 62908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 516.803470] LNetError: 62908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 516.812467] LNet: Removed LNI 192.168.203.37@tcp [ 516.962589] Key type .llcrypt unregistered [ 516.964042] Key type ._llcrypt unregistered [ 522.737714] Key type ._llcrypt registered [ 522.739693] Key type .llcrypt registered [ 523.248885] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:34:41 (1789317281) [ 523.563273] Key type .llcrypt unregistered [ 523.564967] Key type ._llcrypt unregistered [ 525.677464] Key type ._llcrypt registered [ 525.679097] Key type .llcrypt registered [ 525.755934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 525.762023] alg: No test for adler32 (adler32-zlib) [ 526.572660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 526.583285] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 529.661746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 532.877365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.892957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 532.906474] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 533.467347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e remote_dropped -r 1 [ 533.488163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e remote_dropped -r 1 [ 533.506347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e remote_dropped -r 1 [ 533.526387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e remote_dropped -r 1 [ 533.547124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e remote_dropped -r 1 [ 533.567477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e remote_dropped -r 1 [ 533.588153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e remote_dropped -r 1 [ 533.610666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e remote_dropped -r 1 [ 533.627904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 533.638603] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 533.641358] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.137@tcp [ 533.731300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 533.750204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 533.766437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 533.779625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 533.794341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 533.809563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 533.830465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 533.847863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 533.865708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 533.877048] LNet: 64366:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 533.885829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 533.900884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 533.920177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 533.940690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 533.959220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 533.975098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 533.992410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 534.008277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 534.228967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e remote_error -r 1 [ 534.250322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e remote_error -r 1 [ 534.271530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e remote_error -r 1 [ 534.290383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e remote_error -r 1 [ 534.308960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e remote_error -r 1 [ 534.329909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e remote_error -r 1 [ 534.348730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e remote_error -r 1 [ 534.366529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e remote_error -r 1 [ 534.385094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 534.395493] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 534.398763] LNet: Skipped 2 previous similar messages [ 534.483131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 534.497650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 534.514106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 534.530292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 534.545228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 534.560683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 534.574697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 534.590141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 534.605790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 534.615924] LNet: 64491:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 534.625461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 534.641657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 534.657454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 534.674693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 534.689823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 534.706778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 534.723570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 534.739463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 534.954696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e remote_timeout -r 1 [ 534.973924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e remote_timeout -r 1 [ 534.992168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e remote_timeout -r 1 [ 535.010607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e remote_timeout -r 1 [ 535.028375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e remote_timeout -r 1 [ 535.048239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e remote_timeout -r 1 [ 535.067372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e remote_timeout -r 1 [ 535.086534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e remote_timeout -r 1 [ 535.105209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.137@tcp [ 535.208327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 535.223309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 535.238877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 535.255883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 535.271225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 535.286400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 535.300144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 535.315103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 535.330935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 535.345836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 535.362497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 535.378586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 535.393715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 535.410770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 535.429159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 535.448106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 535.464040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 537.838396] LNet: 64694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.842210] LNetError: 64694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.846301] LNet: Removed LNI 192.168.203.37@tcp [ 537.854346] LNet: Removed LNI 192.168.203.37@tcp1 [ 538.025472] Key type .llcrypt unregistered [ 538.026874] Key type ._llcrypt unregistered [ 543.933746] Key type ._llcrypt registered [ 543.934869] Key type .llcrypt registered [ 544.474990] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:35:02 (1789317302) [ 545.100236] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 545.749817] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:35:04 (1789317304) [ 546.053469] Key type .llcrypt unregistered [ 546.054987] Key type ._llcrypt unregistered [ 548.339522] Key type ._llcrypt registered [ 548.341180] Key type .llcrypt registered [ 548.421845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 548.429430] alg: No test for adler32 (adler32-zlib) [ 549.235294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 549.247822] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 552.418246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 552.692899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e network_timeout -r 1 [ 552.709177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e network_timeout -r 1 [ 552.728361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 552.738411] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 552.824934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 552.841735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 552.857404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 552.872155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 552.888223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 552.902380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 552.916458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 552.931821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 552.947241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 552.956652] LNet: 66230:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 552.967624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 552.984620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 553.000157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 553.014323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 553.027514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 553.042521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 553.152576] LNet: 66269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.156114] LNetError: 66269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 553.163381] LNet: Removed LNI 192.168.203.37@tcp [ 553.326543] Key type .llcrypt unregistered [ 553.328727] Key type ._llcrypt unregistered [ 559.526317] Key type ._llcrypt registered [ 559.527774] Key type .llcrypt registered [ 559.600473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 559.608869] alg: No test for adler32 (adler32-zlib) [ 560.421870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 560.435667] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 563.501905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 566.716638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.732319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 566.741898] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 567.280516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e network_timeout -r 1 [ 567.298209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e network_timeout -r 1 [ 567.315100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e network_timeout -r 1 [ 567.332975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e network_timeout -r 1 [ 567.349671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e network_timeout -r 1 [ 567.366417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e network_timeout -r 1 [ 567.384649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e network_timeout -r 1 [ 567.401805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e network_timeout -r 1 [ 567.418990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 567.427475] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 567.522289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 567.534641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 567.549709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 567.566151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 567.581693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 567.595467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 567.608490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 567.622665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 567.637793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 567.647108] LNet: 1 local NIs in recovery (showing 1): 192.168.203.37@tcp [ 567.648282] LNet: 67141:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 567.649854] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.137@tcp [ 567.659480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 567.676390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 567.694970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 567.714387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 567.730493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 567.747275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 569.920903] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 570.040876] LNet: 67214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.044136] LNetError: 67214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 570.048843] LNet: Removed LNI 192.168.203.37@tcp [ 570.054500] LNet: Removed LNI 192.168.203.37@tcp1 [ 570.211179] Key type .llcrypt unregistered [ 570.213167] Key type ._llcrypt unregistered [ 576.140874] Key type ._llcrypt registered [ 576.142534] Key type .llcrypt registered [ 576.676860] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:35:34 (1789317334) [ 576.812201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 576.818957] alg: No test for adler32 (adler32-zlib) [ 577.634090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.658822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 577.670604] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 577.977539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 577.989686] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 578.334138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 578.366094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 578.387450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 578.407782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 578.427913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 578.444803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 578.464376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.37@tcp [ 578.475254] LNetError: 68309:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.37@tcp to recovery [ 578.655279] LNet: 1 local NIs in recovery (showing 1): 192.168.203.37@tcp [ 578.655325] LNet: There was an unexpected network error while writing to 192.168.203.37: rc = -22 [ 580.703398] LNet: There was an unexpected network error while writing to 192.168.203.37: rc = -22 [ 581.659969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 584.800684] LNet: There was an unexpected network error while writing to 192.168.203.37: rc = -22 [ 584.881606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 584.899991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 584.929750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.940094] LNet: 68416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.944423] LNetError: 68416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.947826] LNet: Removed LNI 192.168.203.37@tcp [ 584.960561] LNet: Removed LNI 192.168.203.37@tcp1 [ 584.970236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.995722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.010059] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 585.313658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 585.325456] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 585.679106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 585.695758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 585.725819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 585.749166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 585.766877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 585.782865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 585.800133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 585.816291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.37@tcp [ 585.826459] LNetError: 68688:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.37@tcp to recovery [ 588.063266] LNet: There was an unexpected network error while writing to 192.168.203.37: rc = -22 [ 588.066888] LNet: Skipped 1 previous similar message [ 589.007402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 592.160223] LNet: There was an unexpected network error while writing to 192.168.203.37: rc = -22 [ 596.332193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 596.346623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 596.362778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 599.172726] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:35:57 (1789317357) [ 599.230028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.241045] LNet: 69403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.243872] LNetError: 69403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 599.247491] LNetError: 69403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 599.251565] LNet: Removed LNI 192.168.203.37@tcp [ 599.262415] LNet: Removed LNI 192.168.203.37@tcp1 [ 599.273866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.300445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 599.313202] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 599.610256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 599.622534] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 600.006929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 600.041665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 600.058586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 600.073918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 600.098114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 600.117249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 600.134656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.37@tcp --health 0 [ 600.180329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 600.351380] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.37@tcp [ 601.375608] LNet: There was an unexpected network error while writing to 192.168.203.37: rc = -22 [ 601.378888] LNet: Skipped 1 previous similar message [ 616.709996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 616.765712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 616.781196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 616.795701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.37@tcp --health 500 [ 616.834549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 623.050419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 623.076279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.085944] LNet: 69911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.089710] LNetError: 69911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.092400] LNetError: 69911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 623.095475] LNet: Removed LNI 192.168.203.37@tcp [ 623.103389] LNet: Removed LNI 192.168.203.37@tcp1 [ 623.112566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.136664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 623.148171] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 623.406945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 623.415360] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 623.727580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 623.743239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 623.768970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 623.783633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 623.798669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 623.816658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 623.833857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 623.853060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.37@tcp --health 0 [ 624.159325] LNet: There was an unexpected network error while writing to 192.168.203.37: rc = -22 [ 624.162352] LNet: Skipped 3 previous similar messages [ 624.970655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 635.387263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 635.401937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 635.416638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 637.680935] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:36:36 (1789317396) [ 637.815057] LNet: 70967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.818474] LNetError: 70967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.821522] LNetError: 70967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 637.825838] LNet: Removed LNI 192.168.203.37@tcp [ 637.834551] LNet: Removed LNI 192.168.203.37@tcp1 [ 638.075701] Key type .llcrypt unregistered [ 638.077247] Key type ._llcrypt unregistered [ 640.176261] Key type ._llcrypt registered [ 640.177804] Key type .llcrypt registered [ 640.260532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 640.266971] alg: No test for adler32 (adler32-zlib) [ 641.083851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 641.094984] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 644.704559] Lustre: DEBUG MARKER: Initial discovery [ 644.724566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.137@tcp [ 645.635503] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 645.665831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 645.684788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.137@tcp [ 645.696953] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 645.708175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 645.723937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.137@tcp -r 1 -m GET -l 3 [ 645.739441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.137@tcp -r 1 -m GET -e local_error [ 645.756596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.137@tcp -r 1 -m PUT -l 6 [ 646.239238] LNet: 1 local NIs in recovery (showing 1): 192.168.203.37@tcp [ 646.281288] Lustre: DEBUG MARKER: Force oleg337-client.virtnet to discover 192.168.203.137@tcp (in background) [ 646.308138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.137@tcp [ 646.842905] Lustre: DEBUG MARKER: Force oleg337-server to discover 192.168.203.37@tcp [ 647.719063] Lustre: DEBUG MARKER: Wait for 71718 [ 649.375273] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 649.902540] Lustre: DEBUG MARKER: Finished wait on 71718 [ 649.935952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 649.956200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 650.074112] LNet: 72175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.077353] LNetError: 72175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 650.087382] LNet: Removed LNI 192.168.203.37@tcp [ 650.236585] Key type .llcrypt unregistered [ 650.238222] Key type ._llcrypt unregistered [ 655.902936] Key type ._llcrypt registered [ 655.904407] Key type .llcrypt registered [ 656.415377] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:36:54 (1789317414) [ 656.582387] Key type .llcrypt unregistered [ 656.583434] Key type ._llcrypt unregistered [ 658.598975] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 658.665283] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 658.667520] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 658.670452] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 660.863336] Key type ._llcrypt registered [ 660.864417] Key type .llcrypt registered [ 660.937601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 660.942876] alg: No test for adler32 (adler32-zlib) [ 661.755980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.779065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.790363] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 662.037883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 662.047087] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 665.002471] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:37:03 (1789317423) [ 665.034276] LNet: 74237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.037433] LNetError: 74237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 665.041165] LNet: Removed LNI 10.1.2.3@tcp [ 665.049850] LNet: Removed LNI 192.168.203.37@tcp [ 665.315664] Key type .llcrypt unregistered [ 665.317174] Key type ._llcrypt unregistered [ 667.481457] Key type ._llcrypt registered [ 667.482858] Key type .llcrypt registered [ 667.550899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 667.557990] alg: No test for adler32 (adler32-zlib) [ 668.367417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.396543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 668.408699] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 668.678541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 668.688521] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 669.077257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.37@tcp [ 670.111119] LNetError: Unexpected error -22 connecting to 192.168.203.37@tcp at host 192.168.203.37:988 [ 671.436834] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:37:09 (1789317429) [ 671.471948] LNet: 75364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.475618] LNetError: 75364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 671.479328] LNet: Removed LNI 10.1.2.3@tcp [ 671.485922] LNet: Removed LNI 192.168.203.37@tcp [ 671.739582] Key type .llcrypt unregistered [ 671.741368] Key type ._llcrypt unregistered [ 673.864642] Key type ._llcrypt registered [ 673.866131] Key type .llcrypt registered [ 673.932583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 673.939137] alg: No test for adler32 (adler32-zlib) [ 674.753202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.776854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 674.788074] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 675.076876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 675.086746] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 675.439261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.37@tcp1 --nid 192.168.203.37@tcp2 [ 677.129042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.37@tcp1 192.168.203.37@tcp2 [ 677.141387] LNetError: 76201:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.37@tcp1 to 192.168.203.37@tcp2 - Route table may be misconfigured [ 677.153382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.37@tcp2 192.168.203.37@tcp1 [ 679.553291] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:37:17 (1789317437) [ 679.610182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.621961] LNet: 76772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.625352] LNetError: 76772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.629666] LNet: Removed LNI 192.168.203.37@tcp1 [ 680.677548] LNet: Removed LNI 192.168.203.37@tcp2 [ 680.687331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.713237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 680.724913] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 681.038555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 681.048882] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 681.377890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 681.396944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.37@tcp 192.168.203.37@tcp [ 681.414618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.37@tcp1 192.168.203.37@tcp1 [ 681.432804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.37@tcp -d 192.168.203.37@tcp -e network_timeout [ 681.450509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -e network_timeout [ 681.467571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -e network_timeout [ 681.485725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -e network_timeout [ 681.501483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 681.510599] LNet: There was an unexpected network error while writing to 192.168.203.37: rc = -22 [ 681.520041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 681.557738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 681.595807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 681.759104] LNet: 1 local NIs in recovery (showing 1): 192.168.203.37@tcp [ 683.850170] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:37:22 (1789317442) [ 683.884994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.897753] LNet: 77628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.902115] LNetError: 77628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.908061] LNetError: 77628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 683.912835] LNet: Removed LNI 192.168.203.37@tcp [ 684.967752] LNet: Removed LNI 192.168.203.37@tcp1 [ 684.977519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.015384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 685.129521] LNet: 77672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.290425] Key type .llcrypt unregistered [ 685.292040] Key type ._llcrypt unregistered [ 689.122061] Key type ._llcrypt registered [ 689.123440] Key type .llcrypt registered [ 689.614323] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:37:27 (1789317447) [ 689.740534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 689.748175] alg: No test for adler32 (adler32-zlib) [ 690.564895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.145865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 691.746595] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:37:30 (1789317450) [ 691.776507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.787493] LNet: 78760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.797495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.822540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.831513] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 692.095699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 692.104762] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 692.452445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 692.468905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp1 [ 692.486377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp1 [ 694.851889] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:37:33 (1789317453) [ 695.527350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg337-server [ 696.115596] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:37:34 (1789317454) [ 696.805285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg337-server [ 697.415902] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:37:35 (1789317455) [ 698.084584] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg337-server [ 698.712474] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:37:37 (1789317457) [ 699.329111] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg337-server [ 699.902773] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:37:38 (1789317458) [ 700.551431] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg337-server [ 701.114666] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:37:39 (1789317459) [ 701.702047] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg337-server [ 702.262966] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:37:40 (1789317460) [ 702.898744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg337-server [ 703.472196] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:37:41 (1789317461) [ 704.076087] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg337-server [ 704.633825] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:37:43 (1789317463) [ 705.263538] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg337-server [ 705.882517] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:37:44 (1789317464) [ 705.918174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.928143] LNet: 82634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.931317] LNetError: 82634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.934285] LNet: Removed LNI 192.168.203.37@tcp [ 706.981687] LNet: Removed LNI 192.168.203.37@tcp1 [ 706.990343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.014610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 707.027599] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 707.320347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 707.351737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 708.452221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.461342] LNet: 82794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.464818] LNetError: 82794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.469080] LNetError: 82794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 708.480666] LNet: Removed LNI 192.168.203.37@tcp [ 708.489602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.515990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 708.530034] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 708.837641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 708.901652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.925631] LNet: Removed LNI 192.168.203.37@tcp [ 709.935429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.960684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 709.973127] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 710.273190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 710.310236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 710.389101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.398829] LNetError: 83087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.403563] LNetError: 83087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 710.414265] LNet: Removed LNI 192.168.203.37@tcp [ 710.422786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.447716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 710.459139] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 710.758743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 710.835314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.845465] LNet: 83232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.848648] LNet: 83232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 711.909254] LNet: Removed LNI 192.168.203.37@tcp [ 711.916482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.937643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 711.947464] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 712.211057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 712.242873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 712.308840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.381291] LNet: Removed LNI 192.168.203.37@tcp [ 713.388329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.407291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 713.419773] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 713.692521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 713.762135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.770711] LNetError: 83525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.773764] LNetError: 83525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 713.782360] LNet: Removed LNI 192.168.203.37@tcp [ 713.790761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.812325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.822306] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 714.078186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 714.108341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 715.225357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.233235] LNet: 83681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.236415] LNet: 83681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 716.261390] LNet: Removed LNI 192.168.203.37@tcp [ 716.269360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.292452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 716.304639] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 716.596926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 716.672916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.733290] LNet: Removed LNI 192.168.203.37@tcp [ 717.741579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.761594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.771108] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 718.020817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 718.049676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 718.137356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.147313] LNetError: 83973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.150443] LNetError: 83973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 718.164328] LNet: Removed LNI 192.168.203.37@tcp [ 718.173395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.196886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 718.208858] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 718.491826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 719.615250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.676631] LNet: Removed LNI 192.168.203.37@tcp [ 720.685503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.708700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 720.718127] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 721.005507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 721.037632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 721.129296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.149684] LNet: Removed LNI 192.168.203.37@tcp [ 722.158405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.184888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 722.196734] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 722.490372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 722.587532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.621303] LNet: Removed LNI 192.168.203.37@tcp [ 723.632945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.660723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 723.672064] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 723.970269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 723.997445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.008430] LNet: 84552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.011629] LNet: 84552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 724.026491] LNet: Removed LNI 192.168.203.37@tcp [ 724.039437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.065681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 724.078204] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 724.384370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 724.415606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.433560] LNet: Removed LNI 192.168.203.37@tcp [ 724.443564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.468806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 724.505360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.521213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.544557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 724.576923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.592588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.617408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 724.649672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.668731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.692355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 724.705720] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 725.016503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.030332] LNet: Removed LNI 192.168.203.37@tcp [ 725.039528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.068485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 727.583455] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:38:05 (1789317485) [ 727.620471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.638916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.667334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 727.678413] LNet: Added LNI 192.168.203.37@tcp231 [8/256/0/180] [ 728.168460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.177835] LNetError: 85579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.180968] LNetError: 85579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 728.197321] LNet: Removed LNI 192.168.203.37@tcp231 [ 728.205899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.235346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 728.247136] LNet: Added LNI 192.168.203.37@tcp231 [8/256/0/180] [ 728.250121] LNet: 85587:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 728.421309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 728.439143] LNet: Removed LNI 192.168.203.37@tcp231 [ 728.448935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 728.465675] LNet: Added LNI 192.168.203.37@tcp231 [8/256/0/180] [ 728.918863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.938612] LNet: Removed LNI 192.168.203.37@tcp231 [ 728.947567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.976794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 728.988249] LNet: Added LNI 192.168.203.37@tcp231 [8/256/0/0] [ 728.990920] LNet: 85739:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 729.149589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 729.173415] LNet: Removed LNI 192.168.203.37@tcp231 [ 729.181349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 729.194275] LNet: Added LNI 192.168.203.37@tcp231 [8/256/0/0] [ 729.647958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.666481] LNet: Removed LNI 192.168.203.37@tcp231 [ 729.676361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.706742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 729.719311] LNet: Added LNI 192.168.203.37@tcp231 [8/256/0/60] [ 729.880612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 729.902347] LNet: Removed LNI 192.168.203.37@tcp231 [ 729.911172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 729.923878] LNet: Added LNI 192.168.203.37@tcp231 [8/256/0/60] [ 730.367848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.383498] LNet: Removed LNI 192.168.203.37@tcp231 [ 730.393602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.424819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 730.438245] LNet: Added LNI 192.168.203.37@tcp231 [8/256/0/180] [ 730.441151] LNet: 86041:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 730.444461] LNet: 86041:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 730.606167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 730.627547] LNet: Removed LNI 192.168.203.37@tcp231 [ 730.635230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 730.647019] LNet: Added LNI 192.168.203.37@tcp231 [8/256/0/180] [ 733.289118] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:38:11 (1789317491) [ 733.343346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 733.362731] LNet: Removed LNI 192.168.203.37@tcp231 [ 733.370705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.398375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 733.411218] LNet: Added LNI 192.168.203.37@tcp232 [8/256/0/180] [ 733.687812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 733.703809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 733.720546] LNet: Removed LNI 192.168.203.37@tcp232 [ 733.728914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 733.740409] LNet: Added LNI 192.168.203.37@tcp232 [8/256/0/180] [ 734.018841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 734.036565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 734.058490] LNet: Removed LNI 192.168.203.37@tcp232 [ 734.065950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 734.076480] LNet: Added LNI 192.168.203.37@tcp232 [8/256/0/180] [ 734.349555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 734.364786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 734.380685] LNet: Removed LNI 192.168.203.37@tcp232 [ 734.387494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 734.398862] LNet: Added LNI 192.168.203.37@tcp232 [8/256/0/180] [ 734.658878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 734.674814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 734.693519] LNet: Removed LNI 192.168.203.37@tcp232 [ 734.700518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 736.954911] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:38:15 (1789317495) [ 737.369231] Key type .llcrypt unregistered [ 737.370903] Key type ._llcrypt unregistered [ 739.469814] Key type ._llcrypt registered [ 739.471462] Key type .llcrypt registered [ 739.548076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 739.555647] alg: No test for adler32 (adler32-zlib) [ 740.363412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 740.375864] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 743.451794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 746.384810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.401228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 746.412192] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 746.926075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_interrupt -r 2 [ 746.941162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_interrupt -r 2 [ 746.958451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_interrupt -r 2 [ 746.974532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_interrupt -r 2 [ 746.991800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_interrupt -r 2 [ 747.009290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_interrupt -r 2 [ 747.027580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_interrupt -r 2 [ 747.045124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_interrupt -r 2 [ 747.063365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.37@tcp 192.168.203.137@tcp [ 747.074936] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 747.078269] LNet: 1 local NIs in recovery (showing 1): 192.168.203.37@tcp [ 747.171378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 747.185710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 747.201439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 747.217528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 747.232204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 747.246379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 747.260639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 747.276451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 747.289551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 747.299718] LNet: 88535:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 747.308470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 747.324751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 747.339626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 747.353825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 747.368207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 747.381825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 747.562586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_dropped -r 2 [ 747.579203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_dropped -r 2 [ 747.598107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_dropped -r 2 [ 747.615870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_dropped -r 2 [ 747.632476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_dropped -r 2 [ 747.650649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_dropped -r 2 [ 747.666389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_dropped -r 2 [ 747.682239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_dropped -r 2 [ 747.698559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.37@tcp 192.168.203.137@tcp [ 747.708971] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 747.791735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 747.805378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 747.819168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 747.833079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 747.847542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 747.862779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 747.878130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 747.893968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 747.908595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 747.917051] LNet: 88658:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 747.926167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 747.942673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 747.957338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 747.971974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 747.986970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 748.001337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 748.200078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_aborted -r 2 [ 748.217222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_aborted -r 2 [ 748.238639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_aborted -r 2 [ 748.258290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_aborted -r 2 [ 748.276897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_aborted -r 2 [ 748.295535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_aborted -r 2 [ 748.314355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_aborted -r 2 [ 748.334885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_aborted -r 2 [ 748.355316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.37@tcp 192.168.203.137@tcp [ 748.456629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 748.472427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 748.490453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 748.505116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 748.520112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 748.534968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 748.552292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 748.570670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 748.586351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 748.602543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 748.617649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 748.631675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 748.645964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 748.661723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 748.676325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 748.888257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_no_route -r 2 [ 748.904984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_no_route -r 2 [ 748.922215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_no_route -r 2 [ 748.939404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_no_route -r 2 [ 748.958216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_no_route -r 2 [ 748.975742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_no_route -r 2 [ 748.993901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_no_route -r 2 [ 749.010837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_no_route -r 2 [ 749.026869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.37@tcp 192.168.203.137@tcp [ 749.037529] LNet: There was an unexpected network error while writing to 192.168.203.137: rc = -22 [ 749.040780] LNet: Skipped 2 previous similar messages [ 749.130657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 749.146254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 749.162855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 749.177293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 749.190299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 749.206505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 749.223785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 749.240262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 749.255193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 749.263631] LNet: 88902:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 749.267376] LNet: 88902:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 749.276111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 749.290338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 749.305444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 749.320333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 749.334978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 749.350355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 749.555584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e local_timeout -r 2 [ 749.577482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e local_timeout -r 2 [ 749.596272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e local_timeout -r 2 [ 749.612575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e local_timeout -r 2 [ 749.633484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e local_timeout -r 2 [ 749.652633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e local_timeout -r 2 [ 749.671478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e local_timeout -r 2 [ 749.690599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e local_timeout -r 2 [ 749.710097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.37@tcp 192.168.203.137@tcp [ 749.808204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 749.821371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 749.835532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 749.848632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 749.865308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 749.880791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 749.895592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 749.910642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 749.927468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 749.946626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 749.964221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 749.982658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 749.999893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 750.016359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 750.033058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 750.232861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp -m GET -e remote_dropped -r 2 [ 750.250454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.137@tcp1 -m GET -e remote_dropped -r 2 [ 750.265833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp -m GET -e remote_dropped -r 2 [ 750.283096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp -d 192.168.203.37@tcp1 -m GET -e remote_dropped -r 2 [ 750.300384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp -m GET -e remote_dropped -r 2 [ 750.317140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.137@tcp1 -m GET -e remote_dropped -r 2 [ 750.333619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp -m GET -e remote_dropped -r 2 [ 750.349356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.37@tcp1 -d 192.168.203.37@tcp1 -m GET -e remote_dropped -r 2 [ 750.366645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.37@tcp 192.168.203.137@tcp [ 750.376522] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.137@tcp [ 750.451647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 750.463248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 750.476888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 750.492332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 750.507096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 750.521600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 750.536526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 750.550663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 750.564926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 750.577916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 750.592784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 750.605493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 750.621330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 750.637217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 750.651955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 752.933787] LNet: 89219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.936966] LNetError: 89219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.940252] LNet: Removed LNI 192.168.203.37@tcp [ 752.948587] LNet: Removed LNI 192.168.203.37@tcp1 [ 753.114864] Key type .llcrypt unregistered [ 753.116193] Key type ._llcrypt unregistered [ 758.644116] Key type ._llcrypt registered [ 758.645523] Key type .llcrypt registered [ 759.155296] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:38:37 (1789317517) [ 759.275157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 759.282135] alg: No test for adler32 (adler32-zlib) [ 760.092617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.118525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 760.130447] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 760.467345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 760.483132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.37@tcp [ 760.494426] LNetError: 90192:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.37@tcp to recovery [ 760.531430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.37@tcp [ 760.572872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 762.832501] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:38:41 (1789317521) [ 762.862590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.873510] LNet: 90777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.877517] LNetError: 90777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.886427] LNet: Removed LNI 192.168.203.37@tcp [ 762.896289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.920329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 762.931732] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 763.205125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 763.214948] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 763.586286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.37@tcp [ 763.596072] LNetError: 91048:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.37@tcp to recovery [ 763.935310] LNet: 1 local NIs in recovery (showing 1): 192.168.203.37@tcp [ 763.936354] LNet: There was an unexpected network error while writing to 192.168.203.37: rc = -22 [ 764.679681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.37@tcp [ 769.042737] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:38:47 (1789317527) [ 769.704448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg337-server [ 770.329505] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:38:48 (1789317528) [ 771.017490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg337-server [ 771.656526] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:38:49 (1789317529) [ 772.331930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg337-server [ 772.949646] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:38:51 (1789317531) [ 772.986022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.997401] LNet: 92671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.001369] LNetError: 92671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.005652] LNet: Removed LNI 192.168.203.37@tcp [ 773.011513] LNet: Removed LNI 192.168.203.37@tcp1 [ 773.023321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.047914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.060491] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 773.340237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 773.365754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 773.382415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 773.425771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 773.443434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 773.479244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 775.705517] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:38:54 (1789317534) [ 775.748088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.758118] LNet: 93403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.760597] LNetError: 93403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.762962] LNetError: 93403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 775.771662] LNet: Removed LNI 192.168.203.37@tcp [ 775.779826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.800824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 775.811327] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 778.382952] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:38:56 (1789317536) [ 778.436855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.447639] LNet: 94107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.451686] LNetError: 94107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.459675] LNet: Removed LNI 192.168.203.37@tcp [ 778.467961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.491262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.501762] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 780.816684] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:38:59 (1789317539) [ 781.346091] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 781.966934] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:39:00 (1789317540) [ 782.138575] LNet: Removed LNI 192.168.203.37@tcp [ 782.372638] Key type .llcrypt unregistered [ 782.373923] Key type ._llcrypt unregistered [ 784.328286] Key type ._llcrypt registered [ 784.329294] Key type .llcrypt registered [ 784.390757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.396142] alg: No test for adler32 (adler32-zlib) [ 785.197595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 785.208134] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 788.088969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 789.662985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.137@tcp [ 789.782614] LNet: 95554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.786258] LNetError: 95554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.795521] LNet: Removed LNI 192.168.203.37@tcp [ 789.950399] Key type .llcrypt unregistered [ 789.951746] Key type ._llcrypt unregistered [ 793.809856] Key type ._llcrypt registered [ 793.811338] Key type .llcrypt registered [ 794.308985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 794.895350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 795.522361] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:39:13 (1789317553) [ 796.083982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 796.701503] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:39:15 (1789317555) [ 797.366725] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg337-server [ 797.994932] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:39:16 (1789317556) [ 798.663984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg337-server [ 799.326357] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:39:17 (1789317557) [ 799.818035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.858422] alg: No test for adler32 (adler32-zlib) [ 800.730921] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 800.840323] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 802.431289] Key type lgssc registered [ 802.868605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 811.267585] Key type lgssc unregistered [ 811.392638] LNet: 98718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.397358] LNetError: 98718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.406381] LNet: Removed LNI 192.168.203.37@tcp [ 811.677718] Key type .llcrypt unregistered [ 811.679440] Key type ._llcrypt unregistered [ 817.217459] Key type ._llcrypt registered [ 817.218853] Key type .llcrypt registered [ 817.696060] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:39:36 (1789317576) [ 817.825649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 817.831882] alg: No test for adler32 (adler32-zlib) [ 818.639260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.665143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 818.683536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 818.700942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 818.719494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 820.889850] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:39:39 (1789317579) [ 820.922176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.932604] LNet: 100123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.944090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.971803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.983194] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 821.331380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.340992] LNetError: 100272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.350577] LNet: Removed LNI 192.168.203.37@tcp [ 821.359143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.381951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 821.393128] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 821.567398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.577109] LNet: 100300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.579996] LNet: 100300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 821.589516] LNet: Removed LNI 192.168.203.37@tcp [ 821.597519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.619462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 821.632395] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 823.936097] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:39:42 (1789317582) [ 823.978863] LNet: 100886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.982090] LNetError: 100886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.985626] LNetError: 100886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 823.995835] LNet: Removed LNI 192.168.203.37@tcp [ 824.210597] Key type .llcrypt unregistered [ 824.212179] Key type ._llcrypt unregistered [ 826.257530] Key type ._llcrypt registered [ 826.258950] Key type .llcrypt registered [ 826.324973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 826.331062] alg: No test for adler32 (adler32-zlib) [ 827.142558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.171480] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 827.172972] LNet: *** cfs_fail_loc=e003, val=100*** [ 827.462503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 827.473872] Ping reply received with mapping: [ 827.475567] found metadata reply NI[192.168.203.37@tcp] [ 827.477629] 25 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 827.479743] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 827.481774] 5a 5a 5a 5a 5a 5a 5a 5a [ 827.535832] LNet: 101283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.538997] LNetError: 101283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.553278] LNet: Removed LNI 192.168.203.37@tcp [ 827.776687] Key type .llcrypt unregistered [ 827.778097] Key type ._llcrypt unregistered [ 831.521779] Key type ._llcrypt registered [ 831.523137] Key type .llcrypt registered [ 832.021495] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:39:50 (1789317590) [ 832.205266] Key type .llcrypt unregistered [ 832.206856] Key type ._llcrypt unregistered [ 834.291459] Key type ._llcrypt registered [ 834.293133] Key type .llcrypt registered [ 834.361819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 834.368226] alg: No test for adler32 (adler32-zlib) [ 835.235603] LNetError: 102317:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 835.377640] Key type .llcrypt unregistered [ 835.379535] Key type ._llcrypt unregistered [ 837.152559] Key type ._llcrypt registered [ 837.153947] Key type .llcrypt registered [ 837.629496] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:39:55 (1789317595) [ 837.830395] Key type .llcrypt unregistered [ 837.831976] Key type ._llcrypt unregistered [ 839.918877] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 839.920981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 842.078705] Key type ._llcrypt registered [ 842.079696] Key type .llcrypt registered [ 842.144455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 842.150574] alg: No test for adler32 (adler32-zlib) [ 842.957582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.983970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 846.895387] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:40:05 (1789317605) [ 846.927765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.937866] LNet: 104084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.940071] LNetError: 104084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 846.948774] LNet: Removed LNI 10.1.2.3@tcp [ 846.959973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.228450] Key type .llcrypt unregistered [ 847.229776] Key type ._llcrypt unregistered [ 849.228886] Key type ._llcrypt registered [ 849.230348] Key type .llcrypt registered [ 849.294977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 849.301286] alg: No test for adler32 (adler32-zlib) [ 850.108931] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 850.132976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 850.190061] LNet: 104374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.193214] LNetError: 104374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.204175] LNet: Removed LNI 192.168.203.37@tcp [ 850.466564] Key type .llcrypt unregistered [ 850.467970] Key type ._llcrypt unregistered [ 854.520855] Key type ._llcrypt registered [ 854.522310] Key type .llcrypt registered [ 855.043805] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:40:13 (1789317613) [ 855.272367] Key type .llcrypt unregistered [ 855.273885] Key type ._llcrypt unregistered [ 857.376481] Key type ._llcrypt registered [ 857.378104] Key type .llcrypt registered [ 857.453801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 857.461343] alg: No test for adler32 (adler32-zlib) [ 860.644644] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:40:18 (1789317618) [ 860.678296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.696200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.727939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.738784] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 861.002595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.016759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 861.030571] LNet: Removed LNI 192.168.203.37@tcp [ 861.118354] LNet: 106109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.266417] Key type .llcrypt unregistered [ 861.268021] Key type ._llcrypt unregistered [ 865.021908] Key type ._llcrypt registered [ 865.023300] Key type .llcrypt registered [ 865.512155] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:40:23 (1789317623) [ 865.636640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 865.643091] alg: No test for adler32 (adler32-zlib) [ 866.449097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.472352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.481996] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 866.746886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 866.765740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.37@tcp [ 866.774877] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 866.777239] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 866.778617] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 866, 0, 0 [ 866.780943] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 866.782339] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 866.784496] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 866.786472] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 866, 0, 0 [ 866.788625] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 866.789878] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 866.791773] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 866.793578] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 866, 0, 916 [ 866.796976] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 866.798703] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 866.800780] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 866.802414] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 866, 0, 916 [ 866.805291] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 868.982286] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:40:27 (1789317627) [ 869.140518] LNet: 107644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.145200] LNetError: 107644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.161872] LNet: Removed LNI 192.168.203.37@tcp [ 869.400584] Key type .llcrypt unregistered [ 869.401837] Key type ._llcrypt unregistered [ 871.576296] Key type ._llcrypt registered [ 871.577915] Key type .llcrypt registered [ 871.655596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.663522] alg: No test for adler32 (adler32-zlib) [ 872.473450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 872.485628] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 875.583464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 878.992237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.023266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 879.040401] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 879.486961] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 879.543825] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 879.545985] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 879.548492] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 881.675099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 881.685177] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 885.814264] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:40:44 (1789317644) [ 885.876410] LNet: 109139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.880787] LNetError: 109139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.885197] LNet: Removed LNI 192.168.203.37@tcp [ 886.943141] LNet: Removed LNI 192.168.203.37@tcp1 [ 886.945133] LNetError: 109139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 886.949127] LNetError: 109139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 886.958783] LNet: Removed LNI 10.1.2.3@tcp99 [ 887.215576] Key type .llcrypt unregistered [ 887.217609] Key type ._llcrypt unregistered [ 889.617683] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 889.620318] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.811112] Key type ._llcrypt registered [ 891.812478] Key type .llcrypt registered [ 891.879479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.885435] alg: No test for adler32 (adler32-zlib) [ 892.697022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.723638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 892.736053] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 893.020678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 893.032194] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 893.921741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 893.941786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 893.961575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 893.972751] LNetError: 109886:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 893.987317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 894.007079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 894.167490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 894.204512] LNet: 109906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.208301] LNetError: 109906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 894.211960] LNet: Removed LNI 10.1.2.3@tcp [ 895.269463] LNet: Removed LNI 192.168.203.37@tcp [ 895.503594] Key type .llcrypt unregistered [ 895.504942] Key type ._llcrypt unregistered [ 899.289705] Key type ._llcrypt registered [ 899.291166] Key type .llcrypt registered [ 899.746716] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:40:58 (1789317658) [ 899.868057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.873960] alg: No test for adler32 (adler32-zlib) [ 900.674970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.695371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 900.705296] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 900.958421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 900.986272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg337-client@tcp [ 903.133971] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:41:01 (1789317661) [ 903.182596] LNet: 111432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.186156] LNetError: 111432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.193551] LNet: Removed LNI 192.168.203.37@tcp [ 903.421462] Key type .llcrypt unregistered [ 903.422982] Key type ._llcrypt unregistered [ 905.659527] Key type ._llcrypt registered [ 905.661070] Key type .llcrypt registered [ 905.728810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 905.735299] alg: No test for adler32 (adler32-zlib) [ 906.542804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 906.559449] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 906.571854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 908.993178] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:41:07 (1789317667) [ 909.545408] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 910.139222] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:41:08 (1789317668) [ 910.690215] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 911.267990] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:41:09 (1789317669) [ 911.308812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.317925] LNet: 112851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.320882] LNetError: 112851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.328703] LNet: Removed LNI 192.168.203.37@tcp [ 911.337549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.358847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.369174] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 911.658958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 911.678766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 913.946251] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:41:12 (1789317672) [ 913.993300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.004662] LNet: 113562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.008308] LNetError: 113562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.018445] LNet: Removed LNI 192.168.203.37@tcp [ 914.026227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.047488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 914.059910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 914.311483] Key type .llcrypt unregistered [ 914.313061] Key type ._llcrypt unregistered [ 918.114900] Key type ._llcrypt registered [ 918.116064] Key type .llcrypt registered [ 918.591418] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:41:16 (1789317676) [ 918.748829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.756154] alg: No test for adler32 (adler32-zlib) [ 919.564019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.587451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.599082] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 919.888644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 919.915643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 922.170846] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:41:20 (1789317680) [ 922.198126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.208516] LNet: 115108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.211314] LNetError: 115108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.218511] LNet: Removed LNI 192.168.203.37@tcp [ 922.228628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.253599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 922.268485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 924.412181] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:41:22 (1789317682) [ 924.453777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.464838] LNet: 115690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.479836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.506203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 924.644543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 924.656413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 924.678399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 924.815500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 924.830722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 924.860756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.876572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.910711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 925.050593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 925.064792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 925.090208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 925.242994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 925.256376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 927.668626] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:41:26 (1789317686) [ 927.711163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.725630] LNet: 116322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.728268] LNet: 116322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 927.743888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.768363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.783481] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 928.076959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 928.087574] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 928.436593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 928.465338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 928.547868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.558586] LNetError: 116604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.561106] LNet: Removed LNI 192.168.203.37@tcp [ 928.568503] LNet: Removed LNI 192.168.203.37@tcp2 [ 928.579676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.608527] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 928.612322] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 928.615320] LNet: 116610:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 928.807822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 928.892162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.951143] LNet: Removed LNI 192.168.203.37@tcp [ 929.952862] LNetError: 116647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.956456] LNetError: 116647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 929.966544] LNet: Removed LNI 192.168.203.37@tcp2 [ 929.975788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.000310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 930.011437] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 930.318335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 930.327489] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 930.656646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 930.674261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 930.699359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 930.779215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.791032] LNet: 116932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.794709] LNet: 116932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 931.807251] LNet: Removed LNI 192.168.203.37@tcp [ 932.835601] LNet: Removed LNI 192.168.203.37@tcp2 [ 932.844588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.870914] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 932.874567] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 932.877343] LNet: 116938:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 933.071566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 933.151030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.161926] LNetError: 116975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.165302] LNetError: 116975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 933.169689] LNet: Removed LNI 192.168.203.37@tcp [ 934.181415] LNet: Removed LNI 192.168.203.37@tcp2 [ 934.191293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.217324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 934.228579] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 934.497482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 934.507776] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 934.822483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 934.846718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 934.934274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.942738] LNet: 117254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.945376] LNet: 117254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 935.967137] LNet: Removed LNI 192.168.203.37@tcp [ 935.974470] LNet: Removed LNI 192.168.203.37@tcp2 [ 935.983503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.011744] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 936.015057] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 936.018073] LNet: 117261:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 936.203364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 936.286787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.311164] LNet: Removed LNI 192.168.203.37@tcp [ 937.312986] LNetError: 117298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.316179] LNetError: 117298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 937.326281] LNet: Removed LNI 192.168.203.37@tcp2 [ 937.336129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.361932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 937.373454] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 937.658152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 937.668439] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 938.001294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 938.019607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 938.043812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 938.125747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.167160] LNet: Removed LNI 192.168.203.37@tcp [ 940.197284] LNet: Removed LNI 192.168.203.37@tcp2 [ 940.205433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.230105] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 940.233804] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 940.237268] LNet: 117588:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 940.411786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 940.480597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.491472] LNet: Removed LNI 192.168.203.37@tcp [ 941.541519] LNet: Removed LNI 192.168.203.37@tcp2 [ 941.550504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.574238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.584925] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 941.858082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 941.867318] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 942.170774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 942.184159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 942.207647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 942.333408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.391125] LNet: Removed LNI 192.168.203.37@tcp [ 943.399479] LNet: Removed LNI 192.168.203.37@tcp2 [ 943.410081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.438686] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 943.442593] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 943.629818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 943.747337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.755772] LNet: 117980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.759709] LNet: 117980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 944.799162] LNet: Removed LNI 192.168.203.37@tcp [ 944.807302] LNet: Removed LNI 192.168.203.37@tcp2 [ 944.816612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.840355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 944.853367] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 945.111655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 945.121407] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 945.431956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 945.448124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 945.461594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 945.486157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 945.610940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.619691] LNetError: 118281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.622669] LNetError: 118281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 945.626173] LNet: Removed LNI 192.168.203.37@tcp [ 946.661310] LNet: Removed LNI 192.168.203.37@tcp2 [ 946.669580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.693643] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 946.696387] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 946.699433] LNet: 118288:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 946.702485] LNet: 118288:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 946.883441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 949.276147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 949.822065] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:41:48 (1789317708) [ 949.858442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.868132] LNet: Removed LNI 192.168.203.37@tcp [ 950.885502] LNet: Removed LNI 192.168.203.37@tcp2 [ 950.894671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.011292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 951.022469] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 951.314325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 951.323903] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 951.666151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.676083] LNet: Removed LNI 10.1.2.3@tcp [ 951.681319] LNet: Removed LNI 192.168.203.37@tcp [ 951.690487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.718335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 951.729709] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 951.731739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 951.889961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 951.902533] LNet: Removed LNI 192.168.203.37@tcp [ 951.913364] LNet: Removed LNI 10.1.2.3@tcp [ 954.200680] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:41:52 (1789317712) [ 954.249942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.270322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.294312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 954.305702] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 954.592073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 954.607702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 954.625679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 954.640850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 954.655773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 954.670842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 954.688219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 954.703750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 954.719539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 954.735763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 954.750775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 954.766161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 954.828579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 957.051582] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 12:41:55 (1789317715) [ 957.090744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.117537] LNet: Removed LNI 192.168.203.37@tcp [ 958.127275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.151686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 958.162562] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 958.475841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 958.490519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 958.510697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.529764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.546580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.564606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.580447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.596460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.611844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.626861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.641884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.657722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.698485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 958.718035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.734591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.749430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.767239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.781802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.795204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.809510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.825072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.839969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 958.854335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.37@tcp [ 961.102801] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:41:59 (1789317719) [ 961.139043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.150447] LNet: 121410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.152877] LNet: 121410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 962.213475] LNet: Removed LNI 192.168.203.37@tcp [ 962.221261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.254614] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 962.308095] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 962.309885] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 962.312919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 964.440515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.450268] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 964.706992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 964.717324] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.433371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 965.449689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 965.472771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.37@tcp -m PUT -e local_timeout -r 1 [ 965.496612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 965.518117] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 965.520191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 965.598045] LNetError: 121480:0:(lib-move.c:4794:lnet_parse()) 192.168.203.37@tcp, src 192.168.203.37@tcp: Dropping PUT (error -108 looking up sender) [ 966.623676] LNetError: 121939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 966.627304] LNetError: 121939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 966.631214] LNet: Removed LNI 10.1.2.3@tcp [ 967.651422] LNet: Removed LNI 192.168.203.37@tcp [ 967.817024] Key type .llcrypt unregistered [ 967.818098] Key type ._llcrypt unregistered [ 971.694190] Key type ._llcrypt registered [ 971.695467] Key type .llcrypt registered [ 972.207034] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:42:10 (1789317730) [ 972.320539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 972.326403] alg: No test for adler32 (adler32-zlib) [ 973.129808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.269196] LNet: 122787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.505452] Key type .llcrypt unregistered [ 973.506899] Key type ._llcrypt unregistered [ 975.510794] Key type ._llcrypt registered [ 975.511923] Key type .llcrypt registered [ 975.577360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.583040] alg: No test for adler32 (adler32-zlib) [ 976.390616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 976.401886] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 979.358443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 979.423456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 979.654543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.137@tcp [ 979.760582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 979.775878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 979.791906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 979.807582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 979.823588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 979.839135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 979.852842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 979.868073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 979.882745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 979.891561] LNet: 123247:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 979.899770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 979.914112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 979.928235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 979.941664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 979.955283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 979.969507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 980.218394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 980.302222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 980.314632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 980.329353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 980.342852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 980.355798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 980.369575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 980.383447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 980.396984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 980.410938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 980.419268] LNet: 123366:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 980.427892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 980.441801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 980.456168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 980.470351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 980.483883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 980.497636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 980.717152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.137@tcp [ 980.821874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 980.836904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 980.853564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 980.869167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 980.883160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 980.896409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 980.908906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 980.921500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 980.936155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 980.952487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 980.968478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 980.985363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 981.002355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 981.017990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 981.033414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 981.278446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 981.366319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 981.379899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 981.392770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 981.405781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 981.420874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 981.434293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 981.450588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 981.465465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 981.479086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 981.489739] LNet: 123605:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 981.492844] LNet: 123605:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 981.502064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 981.517307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 981.532271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 981.535239] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.137@tcp [ 981.549570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 981.564854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 981.578667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 981.706807] LNet: 123651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.709585] LNetError: 123651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.719359] LNet: Removed LNI 192.168.203.37@tcp [ 981.865823] Key type .llcrypt unregistered [ 981.867228] Key type ._llcrypt unregistered [ 987.443992] Key type ._llcrypt registered [ 987.445481] Key type .llcrypt registered [ 987.949463] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:42:26 (1789317746) [ 988.071347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 988.077672] alg: No test for adler32 (adler32-zlib) [ 988.893333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.066385] LNet: 124534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.325289] Key type .llcrypt unregistered [ 989.326735] Key type ._llcrypt unregistered [ 991.420038] Key type ._llcrypt registered [ 991.421536] Key type .llcrypt registered [ 991.500654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.508529] alg: No test for adler32 (adler32-zlib) [ 992.324616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 992.336903] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 995.372484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.137@tcp [ 998.362341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.375782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 998.386082] LNet: Added LNI 192.168.203.37@tcp1 [8/256/0/180] [ 998.724570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 998.947991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.137@tcp [ 999.048880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 999.065780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 999.081500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 999.095730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 999.111160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 999.123647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 999.136768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 999.150798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 999.165797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 999.174640] LNet: 125183:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 999.183442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 999.199049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 999.214957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 999.230074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 999.244137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 999.257862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 999.499672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 999.520270] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.137@tcp, 192.168.203.137@tcp1 [ 999.597649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 999.612306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 999.629074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 999.646122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 999.663662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 999.681762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 999.696183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 999.709856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 999.724960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 999.735995] LNet: 125302:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 999.745940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 999.762667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 999.778902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 999.794199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 999.812156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 999.827939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1000.062928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.137@tcp [ 1000.165762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1000.181077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1000.198311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1000.214452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1000.229499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1000.243906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1000.258691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1000.274282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1000.289812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1000.307664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1000.322292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1000.336297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1000.349970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1000.365260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1000.380500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1000.618185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1000.700068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1000.712495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1000.725909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1000.741278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1000.756135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1000.770350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1000.784357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1000.799782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1000.814093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1000.823652] LNet: 125540:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1000.827149] LNet: 125540:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1000.836621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1000.851803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1000.867815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1000.885105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1000.901207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1000.917708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1003.179967] LNet: 125621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.182966] LNetError: 125621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.186117] LNet: Removed LNI 192.168.203.37@tcp [ 1003.196416] LNet: Removed LNI 192.168.203.37@tcp1 [ 1003.333926] Key type .llcrypt unregistered [ 1003.335056] Key type ._llcrypt unregistered [ 1008.835239] Key type ._llcrypt registered [ 1008.836717] Key type .llcrypt registered [ 1009.295203] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:42:47 (1789317767) [ 1009.421774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1009.427358] alg: No test for adler32 (adler32-zlib) [ 1010.231054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.255120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1010.265791] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 1010.539743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1010.546933] LNet: Added LNI 192.168.203.37@tcp2 [8/256/0/180] [ 1010.851101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.37@tcp [ 1010.883399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 1011.935141] LNet: Removed LNI 192.168.203.37@tcp [ 1014.215209] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:42:52 (1789317772) [ 1014.747447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1015.325485] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:42:53 (1789317773) [ 1017.795427] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:42:56 (1789317776) [ 1020.214824] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:42:58 (1789317778) [ 1022.695227] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:43:01 (1789317781) [ 1025.146874] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:43:03 (1789317783) [ 1027.591171] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:43:05 (1789317785) [ 1030.047853] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:43:08 (1789317788) [ 1032.549683] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:43:10 (1789317790) [ 1035.010864] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:43:13 (1789317793) [ 1037.516874] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:43:15 (1789317795) [ 1039.966936] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:43:18 (1789317798) [ 1042.476881] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:43:20 (1789317800) [ 1044.826650] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:43:23 (1789317803) [ 1047.236606] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:43:25 (1789317805) [ 1049.890124] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:43:28 (1789317808) [ 1052.391461] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:43:30 (1789317810) [ 1054.919711] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:43:33 (1789317813) [ 1057.465656] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:43:35 (1789317815) [ 1057.497576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.507789] LNet: 137291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.511294] LNetError: 137291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.528117] LNet: Removed LNI 192.168.203.37@tcp2 [ 1057.539302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.588053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1057.597726] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 1058.012134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.021573] LNet: 137439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.024735] LNetError: 137439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.036413] LNet: Removed LNI 192.168.203.37@tcp [ 1058.045046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.070463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1058.082752] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 1060.345342] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:43:38 (1789317818) [ 1062.770375] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:43:41 (1789317821) [ 1065.352629] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:43:43 (1789317823) [ 1067.750645] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:43:46 (1789317826) [ 1070.048620] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:43:48 (1789317828) [ 1072.487408] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:43:50 (1789317830) [ 1074.962271] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:43:53 (1789317833) [ 1077.455175] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:43:55 (1789317835) [ 1079.855658] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:43:58 (1789317838) [ 1082.194065] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:44:00 (1789317840) [ 1084.385982] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:44:02 (1789317842) [ 1086.614867] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:44:05 (1789317845) [ 1089.004670] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 950 sec =========== 12:44:07 (1789317847) [ 1089.120025] LNet: 144893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.123747] LNetError: 144893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.130602] LNet: Removed LNI 192.168.203.37@tcp [ 1089.354821] Key type .llcrypt unregistered [ 1089.356205] Key type ._llcrypt unregistered [ 1093.284341] Key type ._llcrypt registered [ 1093.285663] Key type .llcrypt registered [ 1093.448421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1093.454777] alg: No test for adler32 (adler32-zlib) [ 1094.321298] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 1094.419906] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 1096.007154] Key type lgssc registered [ 1096.425066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1120.173280] Lustre: Mounted lustre-client - version 2.17.58_39_gbf2c64c [ 1122.551537] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1124.711218] Lustre: Unmounted lustre-client [ 1136.365067] Key type lgssc unregistered [ 1136.488643] LNet: 147827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.492469] LNetError: 147827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.500783] LNet: Removed LNI 192.168.203.37@tcp [ 1136.757090] Key type .llcrypt unregistered [ 1136.758073] Key type ._llcrypt unregistered