[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450887085 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002364] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006924] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009130] LSM: Security Framework initializing [ 0.010060] Yama: becoming mindful. [ 0.011040] SELinux: Initializing. [ 0.012070] *** VALIDATE selinux *** [ 0.020537] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025396] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] *** VALIDATE tmpfs *** [ 0.031434] *** VALIDATE proc *** [ 0.033123] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042090] debug: unmapping init [mem 0xffffffffb2659000-0xffffffffb2660fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045743] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052020] ... event mask: 000000070000000f [ 0.053329] rcu: Hierarchical SRCU implementation. [ 0.055508] smp: Bringing up secondary CPUs ... [ 0.056632] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.060524] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129387] node 0 deferred pages initialised in 64ms [ 0.133033] devtmpfs: initialized [ 0.134360] x86/mm: Memory block size: 128MB [ 0.136938] gcov: version magic: 0x41383552 [ 0.138179] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140208] pinctrl core: initialized pinctrl subsystem [ 0.141166] [ 0.141710] ************************************************************* [ 0.142010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143010] ** ** [ 0.144011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.145011] ** ** [ 0.146012] ** This means that this kernel is built to expose internal ** [ 0.147013] ** IOMMU data structures, which may compromise security on ** [ 0.148012] ** your system. ** [ 0.149013] ** ** [ 0.150010] ** If you see this message and you are not debugging the ** [ 0.151010] ** kernel, report this immediately to your vendor! ** [ 0.152010] ** ** [ 0.153010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154011] ************************************************************* [ 0.155785] NET: Registered protocol family 16 [ 0.156437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.159067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.161069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165111] cpuidle: using governor menu [ 0.166693] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.168455] PCI: Using configuration type 1 for base access [ 0.170094] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.178107] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.180029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.183069] cryptd: max_cpu_qlen set to 1000 [ 0.185280] ACPI: Added _OSI(Module Device) [ 0.186093] ACPI: Added _OSI(Processor Device) [ 0.188011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.189008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.192968] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198443] ACPI: Interpreter enabled [ 0.200058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202011] ACPI: Using IOAPIC for interrupt routing [ 0.203092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.206363] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.215975] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.223077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.228408] acpiphp: Slot [2] registered [ 0.230126] acpiphp: Slot [5] registered [ 0.232098] acpiphp: Slot [6] registered [ 0.233139] acpiphp: Slot [3] registered [ 0.235104] acpiphp: Slot [4] registered [ 0.236160] acpiphp: Slot [7] registered [ 0.238106] acpiphp: Slot [8] registered [ 0.239095] acpiphp: Slot [9] registered [ 0.240087] acpiphp: Slot [10] registered [ 0.241123] acpiphp: Slot [11] registered [ 0.243110] acpiphp: Slot [12] registered [ 0.244092] acpiphp: Slot [13] registered [ 0.246122] acpiphp: Slot [14] registered [ 0.248184] acpiphp: Slot [15] registered [ 0.249097] acpiphp: Slot [16] registered [ 0.251122] acpiphp: Slot [17] registered [ 0.252092] acpiphp: Slot [18] registered [ 0.253080] acpiphp: Slot [19] registered [ 0.254096] acpiphp: Slot [20] registered [ 0.256119] acpiphp: Slot [21] registered [ 0.257100] acpiphp: Slot [22] registered [ 0.259202] acpiphp: Slot [23] registered [ 0.261117] acpiphp: Slot [24] registered [ 0.262096] acpiphp: Slot [25] registered [ 0.264114] acpiphp: Slot [26] registered [ 0.265111] acpiphp: Slot [27] registered [ 0.266090] acpiphp: Slot [28] registered [ 0.268099] acpiphp: Slot [29] registered [ 0.269104] acpiphp: Slot [30] registered [ 0.271106] acpiphp: Slot [31] registered [ 0.272148] PCI host bridge to bus 0000:00 [ 0.274020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.276022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.278024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.281056] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.284027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288201] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.290998] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294388] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.303015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.307055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.310019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.312018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.314020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316526] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.319785] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.322044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.324738] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.328945] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.338996] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.343014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.348012] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.357014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.366015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.385017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.401314] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.410014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.415015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.426016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.436994] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.439389] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.442397] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.444413] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.446222] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.453053] iommu: Default domain type: Passthrough [ 0.455431] SCSI subsystem initialized [ 0.457133] ACPI: bus type USB registered [ 0.458130] usbcore: registered new interface driver usbfs [ 0.460088] usbcore: registered new interface driver hub [ 0.462082] usbcore: registered new device driver usb [ 0.463180] pps_core: LinuxPPS API ver. 1 registered [ 0.465013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.468088] PTP clock support registered [ 0.470090] EDAC MC: Ver: 3.0.0 [ 0.472392] PCI: Using ACPI for IRQ routing [ 0.474850] NetLabel: Initializing [ 0.476011] NetLabel: domain hash size = 128 [ 0.477010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.479118] NetLabel: unlabeled traffic allowed by default [ 0.482148] vgaarb: loaded [ 0.484299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.485011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.490000] clocksource: Switched to clocksource kvm-clock [ 0.602246] VFS: Disk quotas dquot_6.6.0 [ 0.603804] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.606358] *** VALIDATE ramfs *** [ 0.607501] *** VALIDATE hugetlbfs *** [ 0.608802] pnp: PnP ACPI init [ 0.610915] pnp: PnP ACPI: found 6 devices [ 0.631411] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.634436] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.636598] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.638507] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.640046] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.642106] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.644690] NET: Registered protocol family 2 [ 0.646948] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.651511] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.654272] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.659356] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.662726] TCP: Hash tables configured (established 65536 bind 65536) [ 0.665709] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.668669] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.671330] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.674231] NET: Registered protocol family 1 [ 0.678262] RPC: Registered named UNIX socket transport module. [ 0.680369] RPC: Registered udp transport module. [ 0.681843] RPC: Registered tcp transport module. [ 0.683306] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.686024] NET: Registered protocol family 44 [ 0.687686] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.689894] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.691684] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.693899] PCI: CLS 0 bytes, default 64 [ 0.696176] Unpacking initramfs... [ 2.113767] debug: unmapping init [mem 0xffff8d7dfcc64000-0xffff8d7dfffcffff] [ 2.117724] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.119840] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.122620] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.619943] Initialise system trusted keyrings [ 2.621089] Key type blacklist registered [ 2.622980] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.633085] zbud: loaded [ 2.636718] *** VALIDATE nfs *** [ 2.638025] *** VALIDATE nfs4 *** [ 2.639226] pstore: using deflate compression [ 2.642451] Platform Keyring initialized [ 2.743493] NET: Registered protocol family 38 [ 2.745576] Key type asymmetric registered [ 2.747449] Asymmetric key parser 'x509' registered [ 2.749597] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.753179] io scheduler mq-deadline registered [ 2.754947] io scheduler kyber registered [ 2.756398] io scheduler bfq registered [ 2.758057] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.760342] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.762608] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.764892] ACPI: Power Button [PWRF] [ 2.770220] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.777143] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.789248] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.816622] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.847588] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.851799] Non-volatile memory driver v1.3 [ 2.853539] Linux agpgart interface v0.103 [ 2.883169] virtio_blk virtio1: [vda] 146200 512-byte logical blocks (74.9 MB/71.4 MiB) [ 2.885978] vda: detected capacity change from 0 to 74854400 [ 2.899116] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.902089] vdb: detected capacity change from 0 to 1073741824 [ 2.907945] libphy: Fixed MDIO Bus: probed [ 2.914532] usbcore: registered new interface driver usbserial_generic [ 2.917270] usbserial: USB Serial support registered for generic [ 2.919743] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.924365] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.926359] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.929028] mousedev: PS/2 mouse device common for all mice [ 2.931552] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.934394] rtc_cmos 00:05: RTC can wake from S4 [ 2.940520] rtc_cmos 00:05: registered as rtc0 [ 2.942092] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.944628] intel_pstate: CPU model not supported [ 2.947337] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.953202] hid: raw HID events driver (C) Jiri Kosina [ 2.954333] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.956280] usbcore: registered new interface driver usbhid [ 2.956286] usbhid: USB HID core driver [ 2.956396] drop_monitor: Initializing network drop monitor service [ 2.956521] Initializing XFRM netlink socket [ 2.968523] NET: Registered protocol family 10 [ 2.971395] Segment Routing with IPv6 [ 2.972774] NET: Registered protocol family 17 [ 2.974996] mpls_gso: MPLS GSO support [ 2.980378] RAS: Correctable Errors collector initialized. [ 2.982406] AVX version of gcm_enc/dec engaged. [ 2.984201] AES CTR mode by8 optimization enabled [ 3.060826] sched_clock: Marking stable (3060808067, 0)->(3974189434, -913381367) [ 3.063579] registered taskstats version 1 [ 3.065680] Loading compiled-in X.509 certificates [ 3.067806] zswap: loaded using pool lzo/zbud [ 3.089297] Key type big_key registered [ 3.101126] Key type encrypted registered [ 3.102745] ima: No TPM chip found, activating TPM-bypass! [ 3.104797] ima: Allocated hash algorithm: sha1 [ 3.106500] ima: No architecture policies found [ 3.108553] evm: Initialising EVM extended attributes: [ 3.110753] evm: security.selinux [ 3.112026] evm: security.ima [ 3.113257] evm: security.capability [ 3.114773] evm: HMAC attrs: 0x1 [ 3.116856] rtc_cmos 00:05: setting system clock to 2026-08-26 12:23:17 UTC (1787746997) [ 3.122473] debug: unmapping init [mem 0xffffffffb3603000-0xffffffffb37fffff] [ 3.125164] debug: unmapping init [mem 0xffffffffb2382000-0xffffffffb2658fff] [ 3.138171] Write protecting the kernel read-only data: 28672k [ 3.140971] debug: unmapping init [mem 0xffffffffb0a03000-0xffffffffb0bfffff] [ 3.143510] debug: unmapping init [mem 0xffffffffb1314000-0xffffffffb13fffff] [ 3.169769] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.176457] systemd[1]: Detected virtualization kvm. [ 3.178243] systemd[1]: Detected architecture x86-64. [ 3.180098] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.204274] systemd[1]: No hostname configured. [ 3.205910] systemd[1]: Set hostname to . [ 3.207759] random: systemd: uninitialized urandom read (16 bytes read) [ 3.209950] systemd[1]: Initializing machine ID from random generator. [ 3.249682] random: ln: uninitialized urandom read (6 bytes read) [ 3.326396] random: systemd: uninitialized urandom read (16 bytes read) [ 3.329518] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.336097] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.341820] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.882128] device-mapper: uevent: version 1.0.3 [ 3.884556] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.612679] virtio_net virtio0 ens2: renamed from eth0 [ 4.725362] scsi host0: ata_piix [ 4.742626] scsi host1: ata_piix [ 4.744843] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.748660] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.472576] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.622491] random: crng init done [ 9.623872] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.050620] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.235385] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.511600] SELinux: Disabled at runtime. [ 11.570847] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.578219] systemd[1]: Detected virtualization kvm. [ 11.579831] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.062936] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.066125] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.070599] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.074443] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.077654] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.087443] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.092739] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ 12.221589] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.501638] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.924329] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.932374] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.021884] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.036572] EDAC sbridge: Ver: 1.1.2 [ 14.275967] Key type dns_resolver registered [ 14.575631] NFS: Registering the id_resolver key type [ 14.577360] Key type id_resolver registered [ 14.578640] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg604-client login: [ 46.906874] libcfs: loading out-of-tree module taints kernel. [ 46.975688] Key type ._llcrypt registered [ 46.977305] Key type .llcrypt registered [ 47.410842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 47.419187] alg: No test for adler32 (adler32-zlib) [ 48.470345] Lustre: Lustre: Build Version: 2.17.57_81_g37bf112 [ 48.843160] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 50.487212] Key type lgssc registered [ 51.644283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.369044] hrtimer: interrupt took 2312182 ns [ 214.600162] Lustre: Mounted lustre-client - version 2.17.57_81_g37bf112 [ 220.124485] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 237.807679] Lustre: DEBUG MARKER: oleg604-client.virtnet: executing check_logdir /tmp/testlogs/ [ 240.105188] Lustre: lustre-OST0000-osc-ffff8d7e44bd6000: disconnect after 23s idle [ 243.410840] Lustre: DEBUG MARKER: oleg604-client.virtnet: executing yml_node [ 249.465362] Lustre: DEBUG MARKER: Client: 2.17.57.81 [ 252.237307] Lustre: DEBUG MARKER: MDS: 2.17.57.81 [ 255.585674] Lustre: DEBUG MARKER: OSS: 2.17.57.81 [ 257.638947] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 26 08:27:30 EDT 2026 [ 278.279546] Lustre: Unmounted lustre-client [ 308.861609] Key type lgssc unregistered [ 309.243350] LNet: 8413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.254425] LNetError: 8413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.270301] LNet: Removed LNI 192.168.206.4@tcp [ 310.073137] Key type .llcrypt unregistered [ 310.075556] Key type ._llcrypt unregistered [ 338.189196] Key type ._llcrypt registered [ 338.193872] Key type .llcrypt registered [ 338.375925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 338.391521] alg: No test for adler32 (adler32-zlib) [ 339.304212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 339.375643] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 339.400725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 343.917549] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 344.213645] LNet: 9302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.219311] LNetError: 9302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.231897] LNet: Removed LNI 192.168.206.4@tcp [ 345.256789] Key type .llcrypt unregistered [ 345.260144] Key type ._llcrypt unregistered [ 351.736484] Key type ._llcrypt registered [ 351.740697] Key type .llcrypt registered [ 353.255412] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:29:05 (1787747345) [ 353.768357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 353.791491] alg: No test for adler32 (adler32-zlib) [ 354.737266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.836649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.859567] LNet: 9719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.879365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.933794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 354.951276] LNet: 9726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 361.813256] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:29:14 (1787747354) [ 362.429428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.462342] LNet: 10306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.498952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.581638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 370.046040] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:29:23 (1787747363) [ 370.318693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.338369] LNet: 10893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.353230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.452950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 377.392630] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:29:30 (1787747370) [ 377.529687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.569348] LNet: 11471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.601716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.727939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 385.303459] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:29:37 (1787747377) [ 385.414076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.449190] LNet: 12057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.472644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.544435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 385.937300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 392.967446] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:29:45 (1787747385) [ 393.090684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.133901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.225202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 393.518664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 393.924424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 400.100757] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:29:52 (1787747392) [ 400.546392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.577067] LNet: 13250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.584788] LNet: 13250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 400.625805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.732893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 407.706053] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:30:00 (1787747400) [ 407.912247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.962363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.025835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 408.064861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 408.566948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 408.607657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 409.056285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 409.122793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 409.508932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 409.533645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 409.946575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 409.987127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 410.433267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 410.474933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 411.022936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 411.055821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 411.408265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 411.466462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 412.237504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 412.291779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 412.777440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 412.830089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 413.327964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 413.374977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 413.887927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 413.921578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 414.387511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 414.494664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 421.845605] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:30:14 (1787747414) [ 422.113434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.150772] LNet: 14517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.160746] LNet: 14517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 422.193882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.455327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 422.538147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 428.579169] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:30:21 (1787747421) [ 428.720807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.764073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.824144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 428.856629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 435.097947] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:30:28 (1787747428) [ 435.420983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.447188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.496391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 435.524429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 445.648167] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:30:37 (1787747437) [ 445.847799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.005035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.235796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 446.395979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 457.863232] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:30:49 (1787747449) [ 458.172330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.210030] LNet: 16864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.220073] LNet: 16864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 458.250298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.322500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 458.347534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 464.867484] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:30:57 (1787747457) [ 465.123269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.170333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.223496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 465.249785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 472.622917] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:31:05 (1787747465) [ 472.738285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.841414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.012152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.357538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 480.269732] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:31:13 (1787747473) [ 480.402860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.471214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.837406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 481.166369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 488.869536] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:31:21 (1787747481) [ 489.023601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.077480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.217559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 489.485187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.072240] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:31:29 (1787747489) [ 496.303879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.386792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.996355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.305404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 503.302671] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:31:36 (1787747496) [ 503.419855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.468923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.569651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.610671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.334299] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:31:43 (1787747503) [ 510.701284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.743108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.801614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.853972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 518.280117] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:31:50 (1787747510) [ 518.413671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.475456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.620630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 525.632157] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:31:58 (1787747518) [ 525.730525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.756464] LNet: 22297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 525.767734] LNet: 22297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 525.813496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.951547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 532.895413] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:32:06 (1787747526) [ 532.990579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.074548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.184177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 533.751364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 539.437161] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:32:12 (1787747532) [ 539.745690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.767578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.805400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 539.834889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 546.797886] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:32:19 (1787747539) [ 547.061214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.107224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.197377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 547.247984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 554.762240] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:32:27 (1787747547) [ 554.994650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.065064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.239765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 555.321887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 563.470347] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:32:36 (1787747556) [ 563.852338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.915380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.037263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 564.122641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 571.222278] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:32:44 (1787747564) [ 571.729442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.802936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.896911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 572.067424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 572.142813] LNetError: 25836:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 572.258616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 572.327150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 579.385302] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:32:52 (1787747572) [ 579.510095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.539334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.580287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6tDZOP9pIA [ 579.645445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6tDZOP9pIA [ 579.688755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6tDZOP9pIA [ 586.341903] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:32:59 (1787747579) [ 586.453166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.502957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.607166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 586.644928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 586.702338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 592.663939] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:33:05 (1787747585) [ 592.903195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.943273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.026366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 593.057682] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 594.582317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 595.502393] Lustre: Lustre: Build Version: 2.17.57_81_g37bf112 [ 597.407206] Key type lgssc registered [ 598.679956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.893947] LNetError: 27598:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.4@tcp99 has no available nets [ 605.021835] Key type lgssc unregistered [ 605.435682] LNetError: 28709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.501853] LNet: Removed LNI 192.168.206.4@tcp [ 607.118203] Key type .llcrypt unregistered [ 607.120983] Key type ._llcrypt unregistered [ 613.858090] Key type ._llcrypt registered [ 613.859975] Key type .llcrypt registered [ 615.070708] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:33:28 (1787747608) [ 615.613354] Key type .llcrypt unregistered [ 615.615821] Key type ._llcrypt unregistered [ 620.961326] Key type ._llcrypt registered [ 620.964272] Key type .llcrypt registered [ 621.150201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 621.169298] alg: No test for adler32 (adler32-zlib) [ 622.195907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.283655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 622.317791] LNet: 29629:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 622.327180] LNet: 29629:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 622.334700] LNet: 29629:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 622.352513] LNet: Message forwarding will be enabled in 2s [ 625.439280] LNet: Message forwarding enabled [ 625.786439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 625.829413] LNet: Message forwarding will stop in 2s to 6s [ 629.537494] LNet: Message forwarding disabled [ 630.280470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 630.304568] LNet: 29727:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 630.314859] LNet: 29727:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 630.321160] LNet: 29727:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 630.332602] LNet: Message forwarding will be enabled in 2s [ 630.487711] LNet: 29740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.129512] Key type .llcrypt unregistered [ 631.133056] Key type ._llcrypt unregistered [ 643.401559] Key type ._llcrypt registered [ 643.404636] Key type .llcrypt registered [ 644.872579] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:33:57 (1787747637) [ 645.395190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 645.441988] alg: No test for adler32 (adler32-zlib) [ 646.355953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.441626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 653.692311] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:34:06 (1787747646) [ 653.817323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.848881] LNet: 31143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.882976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.989534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 654.030552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 654.065607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 654.131310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 654.217953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 654.294877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 654.351321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 654.389707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 654.435370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 654.464270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 654.491867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 654.518796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 654.550830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 654.583885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 654.619683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 654.648434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 654.670940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 654.695697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 654.716910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 654.733921] LNetError: 31191:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 661.450719] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:34:14 (1787747654) [ 661.836461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.860899] LNet: 31763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.877562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.975495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.813749] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:34:22 (1787747662) [ 671.926552] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg604-server [ 673.711985] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:34:26 (1787747666) [ 673.960816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.002347] LNet: 32695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.042044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.124307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.161183] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 675.306341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[5-9]@tcp [ 675.330845] LNet: 32829:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 675.383551] LNetError: 32700:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.6@tcp is being used as a gateway but routing feature is not turned on [ 675.400987] LNetError: 32700:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.6@tcp has gone from up to down [ 681.779866] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:34:34 (1787747674) [ 683.844676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg604-server [ 685.474771] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:34:38 (1787747678) [ 685.668543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.697279] LNet: 33765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.703160] LNetError: 33765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.722226] LNet: Removed LNI 192.168.206.4@tcp [ 685.736327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.769975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.794683] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 686.262725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[5-10/2]@tcp [ 686.293701] LNet: 33883:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 686.304697] LNet: 33883:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 686.321870] LNetError: 33769:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.7@tcp is being used as a gateway but routing feature is not turned on [ 686.350830] LNetError: 33769:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 686.361855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[5-10/2]@tcp [ 686.362942] LNetError: 33769:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.7@tcp has gone from up to down [ 686.362950] LNetError: 33769:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 692.045902] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:34:45 (1787747685) [ 692.432746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.452984] LNet: 34475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.458564] LNetError: 34475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.480522] LNet: Removed LNI 192.168.206.4@tcp [ 692.508532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.599419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 692.671492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.730633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.796606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 692.871238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.926351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.971955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 693.034597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.066534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.126920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 693.196480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.236606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.280047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 693.339731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.374475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.424488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 693.479585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.509779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.550748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 693.662982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.688926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.751238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 693.838410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.886080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.925649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 694.011943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.046165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.114635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.189307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.224374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.295730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 694.364856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.418194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.529477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 701.173097] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:34:53 (1787747693) [ 704.055347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg604-server [ 705.833842] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:34:58 (1787747698) [ 707.697476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg604-server [ 709.962816] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:35:02 (1787747702) [ 710.185455] LNet: 35877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.191533] LNet: 35877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 710.871207] Key type .llcrypt unregistered [ 710.874651] Key type ._llcrypt unregistered [ 718.000503] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 718.242208] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 718.254651] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 718.264488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 720.899940] Key type ._llcrypt registered [ 720.903896] Key type .llcrypt registered [ 721.110580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 721.130741] alg: No test for adler32 (adler32-zlib) [ 722.143390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.194105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.223149] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 722.797671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 722.817786] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 724.883921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 724.912442] LNet: Removed LNI 10.1.2.3@tcp [ 725.130676] LNet: 36601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.138917] LNetError: 36601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.167898] LNet: Removed LNI 192.168.206.4@tcp [ 725.839533] Key type .llcrypt unregistered [ 725.842822] Key type ._llcrypt unregistered [ 737.184431] Key type ._llcrypt registered [ 737.187251] Key type .llcrypt registered [ 738.765906] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:35:31 (1787747731) [ 739.296470] Key type .llcrypt unregistered [ 739.302235] Key type ._llcrypt unregistered [ 744.724665] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 744.902382] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 744.906151] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 744.915634] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 747.496779] Key type ._llcrypt registered [ 747.499654] Key type .llcrypt registered [ 747.675244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 747.694138] alg: No test for adler32 (adler32-zlib) [ 748.626498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.700436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.729871] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 749.341825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 749.361788] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 751.803089] LNet: 38102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.816564] LNetError: 38102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 751.825344] LNet: Removed LNI 10.1.2.3@tcp [ 751.837125] LNet: Removed LNI 192.168.206.4@tcp [ 752.620532] Key type .llcrypt unregistered [ 752.622907] Key type ._llcrypt unregistered [ 764.103206] Key type ._llcrypt registered [ 764.105493] Key type .llcrypt registered [ 765.520534] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:35:58 (1787747758) [ 766.063376] Key type .llcrypt unregistered [ 766.065462] Key type ._llcrypt unregistered [ 771.890856] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 771.998345] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 772.004440] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 772.013376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 774.608703] Key type ._llcrypt registered [ 774.610570] Key type .llcrypt registered [ 774.781850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 774.798282] alg: No test for adler32 (adler32-zlib) [ 775.762616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.809550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 775.827608] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 777.792708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 777.819890] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 779.534749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 779.567879] LNet: Removed LNI 10.1.2.3@tcp [ 779.591226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 779.625932] LNet: Removed LNI 10.1.2.31@tcp [ 779.646736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 779.671567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 780.523812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 780.553854] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 781.587310] LNet: 40028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.591569] LNetError: 40028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 781.599913] LNet: Removed LNI 10.1.2.31@tcp [ 781.617817] LNet: Removed LNI 10.1.2.3@tcp [ 782.326484] Key type .llcrypt unregistered [ 782.330591] Key type ._llcrypt unregistered [ 794.577594] Key type ._llcrypt registered [ 794.579219] Key type .llcrypt registered [ 796.040570] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:36:28 (1787747788) [ 796.511947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.524713] alg: No test for adler32 (adler32-zlib) [ 797.585934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.698870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.732496] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 799.094926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.156725] LNet: 40985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.180119] LNetError: 40985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.206425] LNet: Removed LNI 192.168.206.4@tcp [ 799.264445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.336847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.4@tcp [ 799.376141] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 800.813780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.4@tcp [ 800.879580] LNet: Removed LNI 192.168.206.4@tcp [ 800.995992] LNet: 41120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.036687] Key type .llcrypt unregistered [ 802.040862] Key type ._llcrypt unregistered [ 816.147589] Key type ._llcrypt registered [ 816.155625] Key type .llcrypt registered [ 817.799954] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:36:50 (1787747810) [ 819.911571] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg604-server [ 821.388354] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:36:54 (1787747814) [ 822.049248] Key type .llcrypt unregistered [ 822.053943] Key type ._llcrypt unregistered [ 827.957314] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 828.066585] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 828.070364] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 828.079040] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 830.830696] Key type ._llcrypt registered [ 830.836868] Key type .llcrypt registered [ 831.184038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 831.199589] alg: No test for adler32 (adler32-zlib) [ 832.094381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.166323] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 832.982717] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 834.965297] LNet: 42959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.971903] LNetError: 42959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.976326] LNet: Removed LNI 192.168.206.4@tcp1 [ 834.985957] LNet: Removed LNI 10.1.2.3@tcp2 [ 835.606450] Key type .llcrypt unregistered [ 835.608993] Key type ._llcrypt unregistered [ 845.306730] Key type ._llcrypt registered [ 845.309346] Key type .llcrypt registered [ 846.639386] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:37:19 (1787747839) [ 847.017916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.031480] alg: No test for adler32 (adler32-zlib) [ 847.997844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.051503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.088975] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 849.022788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.048335] LNet: 43932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.055846] LNetError: 43932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.069957] LNet: Removed LNI 192.168.206.4@tcp [ 849.098077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.201637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 849.227332] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 856.325108] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:37:29 (1787747849) [ 856.528372] LNet: 44521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.540374] LNetError: 44521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.590651] LNet: Removed LNI 192.168.206.4@tcp [ 857.257875] Key type .llcrypt unregistered [ 857.261178] Key type ._llcrypt unregistered [ 863.774950] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 864.028542] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 864.034037] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 864.053326] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 866.618313] Key type ._llcrypt registered [ 866.625559] Key type .llcrypt registered [ 866.789625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.801713] alg: No test for adler32 (adler32-zlib) [ 867.759278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.827080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.862220] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 868.632686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 868.646996] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 870.605543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.640181] LNet: 45271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.649943] LNetError: 45271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 870.660355] LNet: Removed LNI 10.1.2.3@tcp [ 870.678691] LNet: Removed LNI 192.168.206.4@tcp [ 870.721686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.825393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 870.879534] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 870.884039] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 871.449803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.467971] LNet: 45298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.472088] LNetError: 45298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 871.481800] LNetError: 45298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 871.494119] LNet: Removed LNI 10.1.2.3@tcp [ 871.511687] LNet: Removed LNI 192.168.206.4@tcp [ 871.541230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.606539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 871.629521] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 871.634838] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 872.053868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 872.085790] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 872.962583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.983550] LNet: 45461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.991434] LNetError: 45461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 872.997285] LNetError: 45461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 873.005659] LNet: Removed LNI 10.1.2.3@tcp [ 873.009808] LNet: Removed LNI 192.168.206.4@tcp [ 873.022310] LNet: Removed LNI 192.168.206.4@tcp2 [ 873.046460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.116595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 873.169239] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 873.180844] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 873.186558] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 879.768957] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:37:52 (1787747872) [ 880.045722] LNet: 46059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.050850] LNetError: 46059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 880.055987] LNetError: 46059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 880.061697] LNet: Removed LNI 10.1.2.3@tcp [ 880.065921] LNet: Removed LNI 192.168.206.4@tcp [ 880.075360] LNet: Removed LNI 192.168.206.4@tcp2 [ 880.727643] Key type .llcrypt unregistered [ 880.729806] Key type ._llcrypt unregistered [ 887.793421] Key type ._llcrypt registered [ 887.795543] Key type .llcrypt registered [ 887.952483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 887.967949] alg: No test for adler32 (adler32-zlib) [ 888.940446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.028460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 889.055517] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 889.922604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.936433] LNet: 46458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.941476] LNetError: 46458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.953937] LNet: Removed LNI 192.168.206.4@tcp [ 889.968732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.012406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 890.037967] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 890.375660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.407139] LNet: Removed LNI 192.168.206.4@tcp [ 890.438616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.490273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 890.510130] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 890.833776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.856308] LNet: 46518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.868637] LNet: 46518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 890.878536] LNetError: 46518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.884957] LNetError: 46518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 890.903755] LNet: Removed LNI 192.168.206.4@tcp [ 890.917520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.971673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 891.003462] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 891.344668] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.423039] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.425220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.427727] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 893.726544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.745691] LNet: 46596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.753614] LNetError: 46596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.766328] LNet: Removed LNI 192.168.206.4@tcp [ 893.785032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.849702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 893.867879] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 900.451905] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:38:13 (1787747893) [ 900.929662] LNet: 47195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.938586] LNetError: 47195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.955946] LNet: Removed LNI 192.168.206.4@tcp [ 901.609148] Key type .llcrypt unregistered [ 901.611574] Key type ._llcrypt unregistered [ 909.966728] Key type ._llcrypt registered [ 909.972170] Key type .llcrypt registered [ 910.251678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 910.286394] alg: No test for adler32 (adler32-zlib) [ 911.362708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.521680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.618073] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 913.345702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.402508] LNet: 47598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.416807] LNetError: 47598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.440937] LNet: Removed LNI 192.168.206.4@tcp [ 913.505204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.666204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 913.714337] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 924.218239] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:38:36 (1787747916) [ 924.720965] LNet: 48190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.731102] LNetError: 48190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.766104] LNet: Removed LNI 192.168.206.4@tcp [ 925.682799] Key type .llcrypt unregistered [ 925.687405] Key type ._llcrypt unregistered [ 933.702810] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 934.074200] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 934.083622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 934.095360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 937.086181] Key type ._llcrypt registered [ 937.087692] Key type .llcrypt registered [ 937.431515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.476137] alg: No test for adler32 (adler32-zlib) [ 938.386796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.440107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 938.457872] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 939.247494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.266784] LNet: 48639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.272436] LNetError: 48639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.298800] LNet: Removed LNI 192.168.206.4@tcp [ 939.324994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.470895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 939.513342] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 946.971646] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:39:00 (1787747940) [ 947.133511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.190171] LNet: 49234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.215112] LNetError: 49234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.256243] LNet: Removed LNI 192.168.206.4@tcp [ 947.288832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.432429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 953.822536] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:39:06 (1787747946) [ 953.962958] LNet: 49810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.665564] Key type .llcrypt unregistered [ 954.667662] Key type ._llcrypt unregistered [ 961.421324] Key type ._llcrypt registered [ 961.425891] Key type .llcrypt registered [ 961.659877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.690430] alg: No test for adler32 (adler32-zlib) [ 962.660913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.725617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.760638] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 963.703701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.735447] LNet: 50223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.745197] LNetError: 50223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.761927] LNet: Removed LNI 192.168.206.4@tcp [ 963.793124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.850860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 963.863323] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 970.056994] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:39:22 (1787747962) [ 970.201282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.249755] LNet: 50809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.269607] LNetError: 50809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.308451] LNet: Removed LNI 192.168.206.4@tcp [ 970.357793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.420078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 976.684335] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:39:29 (1787747969) [ 978.603461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 980.769960] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:39:33 (1787747973) [ 982.516279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 984.406457] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:39:37 (1787747977) [ 986.372513] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 988.183213] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:39:41 (1787747981) [ 990.090762] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 992.268740] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:39:44 (1787747984) [ 993.735760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 995.919154] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:39:48 (1787747988) [ 996.012336] LNet: 52860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.437556] Key type .llcrypt unregistered [ 996.439896] Key type ._llcrypt unregistered [ 1002.024628] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1002.111153] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1002.118761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1002.124396] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1004.690180] Key type ._llcrypt registered [ 1004.698756] Key type .llcrypt registered [ 1004.821134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1004.836225] alg: No test for adler32 (adler32-zlib) [ 1005.844353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.967491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.017043] LNet: Added LNI 192.168.206.4@tcp [8/512/0/180] [ 1006.027585] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1006.035503] LNet: 53191:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1006.187896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.217812] LNet: 53210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.226456] LNetError: 53210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1006.239738] LNet: Removed LNI 10.1.2.3@tcp [ 1006.243509] LNetError: 53210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.263309] LNet: Removed LNI 192.168.206.4@tcp [ 1006.300777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.392877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.425538] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1006.551565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.573093] LNet: 53234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.580795] LNetError: 53234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.599859] LNet: Removed LNI 192.168.206.4@tcp [ 1006.630186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.680544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.696928] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1006.701337] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1006.702890] LNet: 53242:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1006.826543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.862597] LNet: 53260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.872675] LNetError: 53260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1006.883082] LNet: Removed LNI 10.1.2.3@tcp [ 1006.884665] LNetError: 53260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.911703] LNet: Removed LNI 192.168.206.4@tcp [ 1006.941067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.015343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.067152] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1007.075590] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1007.079806] LNet: 53268:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1007.272824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.300756] LNet: 53294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.306944] LNetError: 53294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.312257] LNet: Removed LNI 10.1.2.3@tcp [ 1007.313765] LNetError: 53294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.324830] LNet: Removed LNI 192.168.206.4@tcp [ 1007.350176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.407418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.428657] LNet: Added LNI 192.168.206.4@tcp [8/256/0/182] [ 1007.437753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1007.529748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.551871] LNet: 53322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.558224] LNetError: 53322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.569199] LNet: Removed LNI 10.1.2.3@tcp [ 1007.577485] LNetError: 53322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.618142] LNet: Removed LNI 192.168.206.4@tcp [ 1007.644652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.768628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.804692] LNet: Added LNI 192.168.206.4@tcp [8/256/0/182] [ 1007.815553] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1007.820453] LNet: 53330:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1007.931616] LNet: Removed LNI 10.1.2.3@tcp [ 1007.951222] LNet: Removed LNI 192.168.206.4@tcp [ 1008.589136] Key type .llcrypt unregistered [ 1008.592585] Key type ._llcrypt unregistered [ 1022.366231] Key type ._llcrypt registered [ 1022.371541] Key type .llcrypt registered [ 1024.157921] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 08:40:16 (1787748016) [ 1026.358518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1028.321566] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:40:21 (1787748021) [ 1028.842076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.858291] alg: No test for adler32 (adler32-zlib) [ 1029.859653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.944453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1030.006075] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1030.996486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.4 [ 1031.014372] LNet: 54610:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1031.021613] LNet: Removed LNI 192.168.206.4@tcp [ 1037.735671] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:40:30 (1787748030) [ 1038.245109] LNet: 55181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.261080] LNetError: 55181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.284568] LNet: Removed LNI 192.168.206.4@tcp [ 1039.122575] Key type .llcrypt unregistered [ 1039.125129] Key type ._llcrypt unregistered [ 1045.966567] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1046.426681] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1048.611603] Key type ._llcrypt registered [ 1048.617769] Key type .llcrypt registered [ 1048.930307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1048.973492] alg: No test for adler32 (adler32-zlib) [ 1050.789546] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1059.748864] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:40:52 (1787748052) [ 1059.891136] LNet: 56083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.895912] LNetError: 56083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1059.925683] LNet: Removed LNI 10.1.2.3@tcp [ 1060.888943] Key type .llcrypt unregistered [ 1060.897089] Key type ._llcrypt unregistered [ 1068.530633] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1068.716441] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1068.720943] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1068.724732] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1071.327616] Key type ._llcrypt registered [ 1071.335649] Key type .llcrypt registered [ 1071.469987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1071.481344] alg: No test for adler32 (adler32-zlib) [ 1072.361327] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1079.465501] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:41:12 (1787748072) [ 1081.113980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1082.802664] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:41:15 (1787748075) [ 1082.889171] LNet: 57271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.905396] LNetError: 57271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1082.942062] LNet: Removed LNI 10.1.2.3@tcp [ 1083.551996] Key type .llcrypt unregistered [ 1083.553474] Key type ._llcrypt unregistered [ 1089.597242] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1089.862953] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1089.866809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1089.874972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1091.462425] Key type ._llcrypt registered [ 1091.464467] Key type .llcrypt registered [ 1091.653312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.663587] alg: No test for adler32 (adler32-zlib) [ 1092.634810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.712350] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1101.456976] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:41:34 (1787748094) [ 1101.926233] LNet: 58505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.936631] LNetError: 58505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1101.971904] LNet: Removed LNI 10.1.2.3@tcp [ 1102.783438] Key type .llcrypt unregistered [ 1102.785307] Key type ._llcrypt unregistered [ 1108.166455] Key type ._llcrypt registered [ 1108.171458] Key type .llcrypt registered [ 1108.351831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.378946] alg: No test for adler32 (adler32-zlib) [ 1109.279498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1109.334040] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1116.946168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 1117.924832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_interrupt -r 1 [ 1117.975358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_interrupt -r 1 [ 1118.011758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1118.028357] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1118.222351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.254741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1118.285265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.338971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.387305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.422371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.467495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.500778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.517539] LNet: 58946:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.545790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.560255] LNet: 1 local NIs in recovery (showing 1): 192.168.206.4@tcp [ 1118.577279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.619544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.659663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.709798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.745118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.774938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1119.418927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_dropped -r 1 [ 1119.447179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_dropped -r 1 [ 1119.486593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1119.505467] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1119.701290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.727059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.763895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.796544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.840081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.885218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.919537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.970598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.993570] LNet: 59055:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.018897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.040945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.076886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.122152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.180552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.239464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.276428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.863134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_aborted -r 1 [ 1120.906894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_aborted -r 1 [ 1120.952449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1120.975243] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1120.980798] LNet: Skipped 1 previous similar message [ 1121.179684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.204780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.229895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.256424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.282329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.320763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.346527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.382254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.406559] LNet: 59165:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1121.428068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.460912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.501566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.556538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.604861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.647408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.683605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1122.345669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_no_route -r 1 [ 1122.390134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_no_route -r 1 [ 1122.440836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1122.715904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.752644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.774656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.804392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.848632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.902461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.956877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.001893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.043316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.084187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.123156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.173935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.214416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.258407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.291911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1123.779748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_timeout -r 1 [ 1123.809343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_timeout -r 1 [ 1123.836428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1123.863162] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1123.867864] LNet: Skipped 2 previous similar messages [ 1124.124490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.166406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.208448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.246497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.270150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.298607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.328426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.359069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.377989] LNet: 59383:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.386242] LNet: 59383:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1124.407836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.443414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.488176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.531220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.568685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.616956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.660351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1125.213032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_error -r 1 [ 1125.253413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_error -r 1 [ 1125.287861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1125.577810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.624598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.680920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.722148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.761959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.818534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.863744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.902643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.925761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.958376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.993536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.040393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.084835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.129683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.167518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1126.671280] LNet: 59537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.679208] LNetError: 59537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1126.698916] LNet: Removed LNI 192.168.206.4@tcp [ 1127.138803] Key type .llcrypt unregistered [ 1127.143506] Key type ._llcrypt unregistered [ 1144.788929] Key type ._llcrypt registered [ 1144.792353] Key type .llcrypt registered [ 1146.234266] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:42:19 (1787748139) [ 1147.496821] Key type .llcrypt unregistered [ 1147.500254] Key type ._llcrypt unregistered [ 1153.863321] Key type ._llcrypt registered [ 1153.865406] Key type .llcrypt registered [ 1154.024315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1154.043246] alg: No test for adler32 (adler32-zlib) [ 1154.988580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1155.044851] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1162.803996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 1174.397141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1174.450109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1174.479892] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1176.326831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_interrupt -r 1 [ 1176.385667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_interrupt -r 1 [ 1176.464049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_interrupt -r 1 [ 1176.546489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_interrupt -r 1 [ 1176.595459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_interrupt -r 1 [ 1176.659174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_interrupt -r 1 [ 1176.739095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_interrupt -r 1 [ 1176.791048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_interrupt -r 1 [ 1176.840448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1176.867050] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1176.872561] LNet: 1 local NIs in recovery (showing 1): 192.168.206.4@tcp [ 1177.142721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.164658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1177.188820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.210760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.235564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.272735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.312217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.334280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.354886] LNet: 61009:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1177.381595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.422179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.465216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.500866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.537549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.584091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.615535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.087784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_dropped -r 1 [ 1178.128578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_dropped -r 1 [ 1178.168931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_dropped -r 1 [ 1178.233461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_dropped -r 1 [ 1178.283118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_dropped -r 1 [ 1178.329077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_dropped -r 1 [ 1178.380936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_dropped -r 1 [ 1178.440020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_dropped -r 1 [ 1178.488735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1178.505405] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1178.512394] LNet: Skipped 2 previous similar messages [ 1178.744708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.769782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.802408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.835401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.879047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.907807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.940880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.964950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.979272] LNet: 61130:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.992749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.026318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.061459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.099642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.130894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.155648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.178911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.833692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_aborted -r 1 [ 1179.869592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_aborted -r 1 [ 1179.906775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_aborted -r 1 [ 1179.939385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_aborted -r 1 [ 1179.970634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_aborted -r 1 [ 1179.995361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_aborted -r 1 [ 1180.029061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_aborted -r 1 [ 1180.073273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_aborted -r 1 [ 1180.119133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1180.157780] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1180.168924] LNet: Skipped 2 previous similar messages [ 1180.408614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.435877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.472573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.509601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.539773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.572703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.609487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.638510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.657516] LNet: 61251:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.677268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.710943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.758710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.809206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.860700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.913814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.943549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1181.555165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_no_route -r 1 [ 1181.594820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_no_route -r 1 [ 1181.632860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_no_route -r 1 [ 1181.664346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_no_route -r 1 [ 1181.699889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_no_route -r 1 [ 1181.738549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_no_route -r 1 [ 1181.776155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_no_route -r 1 [ 1181.818261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_no_route -r 1 [ 1181.890758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1182.142211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.167660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.212985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.250872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.312671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.354264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.386651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.443074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.502049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.550810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.596565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.627782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.669184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.727597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.782668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1183.578885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_timeout -r 1 [ 1183.620847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_timeout -r 1 [ 1183.666562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_timeout -r 1 [ 1183.726574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_timeout -r 1 [ 1183.792946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_timeout -r 1 [ 1183.839880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_timeout -r 1 [ 1183.884954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_timeout -r 1 [ 1183.946865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_timeout -r 1 [ 1184.020074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1184.045414] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1184.050887] LNet: Skipped 5 previous similar messages [ 1184.283479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.337732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.383752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.413557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.455886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.490813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.542099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.584084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.607105] LNet: 61494:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.618177] LNet: 61494:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1184.643549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.698385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.748908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.777736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.818797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.902410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.952238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.618764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_error -r 1 [ 1185.654529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_error -r 1 [ 1185.699995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_error -r 1 [ 1185.744918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_error -r 1 [ 1185.786572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_error -r 1 [ 1185.838925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_error -r 1 [ 1185.905664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_error -r 1 [ 1185.964716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_error -r 1 [ 1185.989707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1186.153849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.206630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.232203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.251889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.283618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.312941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.349627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.403366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.444147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.470561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.499200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.536947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.577696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.621696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.691800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1192.689636] LNet: 61695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.697125] LNetError: 61695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.709559] LNet: Removed LNI 192.168.206.4@tcp [ 1192.719844] LNet: Removed LNI 192.168.206.4@tcp1 [ 1193.197555] Key type .llcrypt unregistered [ 1193.201390] Key type ._llcrypt unregistered [ 1213.061701] Key type ._llcrypt registered [ 1213.064945] Key type .llcrypt registered [ 1214.200158] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:43:27 (1787748207) [ 1215.961662] Key type .llcrypt unregistered [ 1215.966071] Key type ._llcrypt unregistered [ 1225.362409] Key type ._llcrypt registered [ 1225.370789] Key type .llcrypt registered [ 1225.813151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1225.845706] alg: No test for adler32 (adler32-zlib) [ 1227.038602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1227.074322] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1237.610982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 1238.702537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e remote_dropped -r 1 [ 1238.780852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e remote_dropped -r 1 [ 1238.844943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1238.857952] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1239.018708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.048445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1239.081042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.119779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.151669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.184537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.225854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.265423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.294620] LNet: 62948:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.310306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.336637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.368758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.402307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.461867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.496896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.549280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1240.247057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e remote_error -r 1 [ 1240.290061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e remote_error -r 1 [ 1240.346641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1240.365841] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1240.582427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.630445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.667089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.698658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.724855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.756842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.776153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.806280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.818866] LNet: 63058:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.831976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.860678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.883483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.914886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.938259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.963320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.984934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1241.605955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e remote_timeout -r 1 [ 1241.637697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e remote_timeout -r 1 [ 1241.686849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1241.709186] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1241.903759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.926781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.945826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.974739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1242.000976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1242.036698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1242.069492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1242.087689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1242.111215] LNet: 63167:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1242.124832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1242.167812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.191808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.211698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.230875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.257963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.294852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1242.553524] LNet: 63212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.558624] LNetError: 63212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1242.573503] LNet: Removed LNI 192.168.206.4@tcp [ 1242.825777] Key type .llcrypt unregistered [ 1242.830031] Key type ._llcrypt unregistered [ 1261.233979] Key type ._llcrypt registered [ 1261.236653] Key type .llcrypt registered [ 1262.745286] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:44:15 (1787748255) [ 1264.089692] Key type .llcrypt unregistered [ 1264.093131] Key type ._llcrypt unregistered [ 1270.413848] Key type ._llcrypt registered [ 1270.418724] Key type .llcrypt registered [ 1270.747968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1270.776384] alg: No test for adler32 (adler32-zlib) [ 1271.726166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1271.775076] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1280.063777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 1290.422910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1290.490121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1290.532322] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1292.497507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e remote_dropped -r 1 [ 1292.540492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e remote_dropped -r 1 [ 1292.577794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e remote_dropped -r 1 [ 1292.624673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e remote_dropped -r 1 [ 1292.677821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e remote_dropped -r 1 [ 1292.718720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e remote_dropped -r 1 [ 1292.769026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e remote_dropped -r 1 [ 1292.834583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e remote_dropped -r 1 [ 1292.898785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1292.933074] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1292.947490] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.104@tcp [ 1293.217157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1293.276173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1293.338423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1293.371647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1293.408894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1293.455676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1293.500516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1293.547143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1293.580727] LNet: 64682:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1293.612930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1293.671038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1293.730699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.781195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.810668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.844463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.892518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1293.936548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1294.415168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e remote_error -r 1 [ 1294.445351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e remote_error -r 1 [ 1294.470847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e remote_error -r 1 [ 1294.498906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e remote_error -r 1 [ 1294.532485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e remote_error -r 1 [ 1294.566136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e remote_error -r 1 [ 1294.591894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e remote_error -r 1 [ 1294.651109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e remote_error -r 1 [ 1294.704540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1294.733757] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1294.737802] LNet: Skipped 2 previous similar messages [ 1294.955389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1294.986526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1295.016205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.057558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.101425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.156547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.212511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.277096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.322983] LNet: 64805:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.360989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1295.406259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1295.442321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1295.501044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1295.540609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1295.615596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1295.671601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1295.708408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.381562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e remote_timeout -r 1 [ 1296.420310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e remote_timeout -r 1 [ 1296.464382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e remote_timeout -r 1 [ 1296.534798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e remote_timeout -r 1 [ 1296.575257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e remote_timeout -r 1 [ 1296.612304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e remote_timeout -r 1 [ 1296.655495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e remote_timeout -r 1 [ 1296.703277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e remote_timeout -r 1 [ 1296.752682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.104@tcp [ 1296.768995] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1296.772555] LNet: Skipped 1 previous similar message [ 1297.017047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.052421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1297.099039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.129897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.159384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.192862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.214802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.237781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.250551] LNet: 64928:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.263397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.287719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.316981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.348811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.373094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.409278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.440371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1297.477423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.745388] LNet: 65010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.757798] LNetError: 65010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1302.765381] LNet: Removed LNI 192.168.206.4@tcp [ 1302.790285] LNet: Removed LNI 192.168.206.4@tcp1 [ 1303.441921] Key type .llcrypt unregistered [ 1303.444279] Key type ._llcrypt unregistered [ 1323.575463] Key type ._llcrypt registered [ 1323.584020] Key type .llcrypt registered [ 1325.134940] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:45:17 (1787748317) [ 1326.812927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1328.739887] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:45:21 (1787748321) [ 1329.760391] Key type .llcrypt unregistered [ 1329.763850] Key type ._llcrypt unregistered [ 1337.566319] Key type ._llcrypt registered [ 1337.569580] Key type .llcrypt registered [ 1337.736628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1337.748708] alg: No test for adler32 (adler32-zlib) [ 1338.786375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1338.827602] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1347.351522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 1348.211310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e network_timeout -r 1 [ 1348.253806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e network_timeout -r 1 [ 1348.298112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 1348.315323] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1348.736331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.811487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1348.865632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.945145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1348.990778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1349.053518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1349.089329] LNet: 1 local NIs in recovery (showing 1): 192.168.206.4@tcp [ 1349.103477] LNet: There was an unexpected network error while writing to 192.168.206.4: rc = -22 [ 1349.115268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1349.194236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1349.227798] LNet: 66560:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1349.267329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1349.316594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1349.353089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.396766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.479772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.549346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.889938] LNet: 66603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.894969] LNetError: 66603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1350.949749] LNet: Removed LNI 192.168.206.4@tcp [ 1351.529327] Key type .llcrypt unregistered [ 1351.530980] Key type ._llcrypt unregistered [ 1372.821341] Key type ._llcrypt registered [ 1372.827360] Key type .llcrypt registered [ 1373.205364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1373.220110] alg: No test for adler32 (adler32-zlib) [ 1374.153224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1374.186399] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1383.979862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 1393.394463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.442669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1393.472420] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1395.253395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e network_timeout -r 1 [ 1395.280394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e network_timeout -r 1 [ 1395.316466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e network_timeout -r 1 [ 1395.351720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e network_timeout -r 1 [ 1395.380859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e network_timeout -r 1 [ 1395.414087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e network_timeout -r 1 [ 1395.454986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e network_timeout -r 1 [ 1395.486597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e network_timeout -r 1 [ 1395.517595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 1395.544773] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1395.680744] LNet: 1 local NIs in recovery (showing 1): 192.168.206.4@tcp [ 1395.695179] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.104@tcp [ 1395.779309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1395.809708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1395.838810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1395.891475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1395.938581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1395.986984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1396.044600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1396.104236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1396.127917] LNet: 67486:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1396.156963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1396.240409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1396.290493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1396.358789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1396.431806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1396.487862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1401.874461] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1402.420375] LNet: 67564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.435845] LNetError: 67564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1402.448173] LNet: Removed LNI 192.168.206.4@tcp [ 1402.476454] LNet: Removed LNI 192.168.206.4@tcp1 [ 1403.005879] Key type .llcrypt unregistered [ 1403.012594] Key type ._llcrypt unregistered [ 1423.723692] Key type ._llcrypt registered [ 1423.727934] Key type .llcrypt registered [ 1425.052029] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:46:57 (1787748417) [ 1425.417189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1425.436423] alg: No test for adler32 (adler32-zlib) [ 1426.295563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.332813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1426.348858] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1427.027628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1427.051113] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1427.932748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 1427.998125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1428.053877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1428.111446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1428.165374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1428.209829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1428.252546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.4@tcp [ 1428.273441] LNetError: 68665:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.4@tcp to recovery [ 1428.383422] LNet: 1 local NIs in recovery (showing 1): 192.168.206.4@tcp [ 1428.383898] LNet: There was an unexpected network error while writing to 192.168.206.4: rc = -22 [ 1430.431846] LNet: There was an unexpected network error while writing to 192.168.206.4: rc = -22 [ 1430.755367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1434.527323] LNet: There was an unexpected network error while writing to 192.168.206.4: rc = -22 [ 1435.497194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1435.553559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1435.620961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.653409] LNet: 68772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.659846] LNetError: 68772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.666434] LNet: Removed LNI 192.168.206.4@tcp [ 1435.677995] LNet: Removed LNI 192.168.206.4@tcp1 [ 1435.716732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.803330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.848151] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1436.643871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1436.666662] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1437.352154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 1437.401945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1437.510535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1437.585781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1437.638692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1437.685723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1437.711138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1437.762587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.4@tcp [ 1437.794856] LNetError: 69045:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.4@tcp to recovery [ 1437.863458] LNet: There was an unexpected network error while writing to 192.168.206.4: rc = -22 [ 1440.285810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1444.000153] LNet: There was an unexpected network error while writing to 192.168.206.4: rc = -22 [ 1444.013473] LNet: Skipped 1 previous similar message [ 1447.245467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1447.288533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1447.326354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1453.903261] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:47:27 (1787748447) [ 1454.062446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.078534] LNet: 69741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.083889] LNetError: 69741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.087940] LNetError: 69741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1455.136883] LNet: Removed LNI 192.168.206.4@tcp [ 1455.143079] LNetError: 69741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.161464] LNet: Removed LNI 192.168.206.4@tcp1 [ 1455.186499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.295770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.340365] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1455.987609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1456.007164] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1456.918627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 1456.983634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1457.046069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1457.111294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1457.159384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1457.194442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1457.244458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.4@tcp --health 0 [ 1457.311184] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.4@tcp [ 1457.391976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1458.335776] LNet: There was an unexpected network error while writing to 192.168.206.4: rc = -22 [ 1458.343429] LNet: Skipped 1 previous similar message [ 1472.785285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1472.942563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1472.960335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1473.001450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.4@tcp --health 500 [ 1473.087932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1480.731068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1480.790298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.808758] LNet: 70241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.816939] LNetError: 70241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.825412] LNet: Removed LNI 192.168.206.4@tcp [ 1480.838597] LNet: Removed LNI 192.168.206.4@tcp1 [ 1480.859598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.928530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.958911] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1481.639371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.671501] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1482.636431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 1482.681543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1482.780402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1482.839061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1482.896452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1482.952182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1482.989162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1483.028381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.4@tcp --health 0 [ 1485.024490] LNet: There was an unexpected network error while writing to 192.168.206.4: rc = -22 [ 1485.035357] LNet: Skipped 3 previous similar messages [ 1485.486754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1495.834180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1495.870426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1495.903120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1505.737677] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:48:18 (1787748498) [ 1506.308673] LNet: 71299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.318714] LNetError: 71299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.325470] LNetError: 71299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1506.333953] LNet: Removed LNI 192.168.206.4@tcp [ 1506.346879] LNet: Removed LNI 192.168.206.4@tcp1 [ 1507.243562] Key type .llcrypt unregistered [ 1507.246812] Key type ._llcrypt unregistered [ 1514.542541] Key type ._llcrypt registered [ 1514.550987] Key type .llcrypt registered [ 1514.803764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1514.856676] alg: No test for adler32 (adler32-zlib) [ 1515.856164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1515.881780] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1527.657720] Lustre: DEBUG MARKER: Initial discovery [ 1527.861333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.104@tcp [ 1531.069967] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1531.150527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1531.227823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.104@tcp [ 1531.268841] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1531.325228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1531.387194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.104@tcp -r 1 -m GET -l 3 [ 1531.443484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.104@tcp -r 1 -m GET -e local_error [ 1531.475663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.104@tcp -r 1 -m PUT -l 6 [ 1532.255923] LNet: 1 local NIs in recovery (showing 1): 192.168.206.4@tcp [ 1533.218407] Lustre: DEBUG MARKER: Force oleg604-client.virtnet to discover 192.168.206.104@tcp (in background) [ 1533.716886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.104@tcp [ 1535.219224] Lustre: DEBUG MARKER: Force oleg604-server to discover 192.168.206.4@tcp [ 1536.805263] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1538.592767] Lustre: DEBUG MARKER: Wait for 72068 [ 1540.128853] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1540.449597] Lustre: DEBUG MARKER: Finished wait on 72068 [ 1540.929097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1540.968373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1541.436466] LNet: 72520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.446427] LNetError: 72520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.469346] LNet: Removed LNI 192.168.206.4@tcp [ 1541.848606] Key type .llcrypt unregistered [ 1541.854516] Key type ._llcrypt unregistered [ 1562.674885] Key type ._llcrypt registered [ 1562.676732] Key type .llcrypt registered [ 1563.899767] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:49:16 (1787748556) [ 1564.583269] Key type .llcrypt unregistered [ 1564.586596] Key type ._llcrypt unregistered [ 1572.600082] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1572.713438] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1572.716509] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1572.721330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1575.447250] Key type ._llcrypt registered [ 1575.459185] Key type .llcrypt registered [ 1575.683928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1575.700399] alg: No test for adler32 (adler32-zlib) [ 1576.750309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.842770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.882897] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1577.740482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1577.759760] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1587.828444] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:49:40 (1787748580) [ 1588.067317] LNet: 74600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.081964] LNetError: 74600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1588.089242] LNet: Removed LNI 10.1.2.3@tcp [ 1588.111907] LNet: Removed LNI 192.168.206.4@tcp [ 1588.797475] Key type .llcrypt unregistered [ 1588.799851] Key type ._llcrypt unregistered [ 1596.612728] Key type ._llcrypt registered [ 1596.616359] Key type .llcrypt registered [ 1596.909491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1596.937627] alg: No test for adler32 (adler32-zlib) [ 1598.074340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.180438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1598.237234] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1599.382353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1599.404802] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1600.842557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.4@tcp [ 1600.911637] LNetError: Unexpected error -22 connecting to 192.168.206.4@tcp at host 192.168.206.4:988 [ 1608.846567] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:50:01 (1787748601) [ 1609.193473] LNet: 75737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.198619] LNetError: 75737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1609.208564] LNet: Removed LNI 10.1.2.3@tcp [ 1609.222178] LNet: Removed LNI 192.168.206.4@tcp [ 1609.946510] Key type .llcrypt unregistered [ 1609.950888] Key type ._llcrypt unregistered [ 1618.439466] Key type ._llcrypt registered [ 1618.444384] Key type .llcrypt registered [ 1618.617183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1618.639532] alg: No test for adler32 (adler32-zlib) [ 1619.525664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.576612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1619.594737] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1620.338746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1620.366275] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 1621.296953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.4@tcp1 --nid 192.168.206.4@tcp2 [ 1626.600224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.4@tcp1 192.168.206.4@tcp2 [ 1626.617685] LNetError: 76577:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.4@tcp1 to 192.168.206.4@tcp2 - Route table may be misconfigured [ 1626.650289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.4@tcp2 192.168.206.4@tcp1 [ 1633.706391] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:50:26 (1787748626) [ 1633.819117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.862320] LNet: 77144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.872934] LNetError: 77144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.883245] LNet: Removed LNI 192.168.206.4@tcp1 [ 1634.921596] LNet: Removed LNI 192.168.206.4@tcp2 [ 1634.976602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.080908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1635.138341] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1635.885589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1635.900164] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1636.650056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 1636.698927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.4@tcp 192.168.206.4@tcp [ 1636.743752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.4@tcp1 192.168.206.4@tcp1 [ 1636.816540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.4@tcp -d 192.168.206.4@tcp -e network_timeout [ 1636.893465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -e network_timeout [ 1636.945900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -e network_timeout [ 1636.982574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -e network_timeout [ 1637.019380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1637.037746] LNet: There was an unexpected network error while writing to 192.168.206.4: rc = -22 [ 1637.056408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1637.091875] LNet: 1 local NIs in recovery (showing 1): 192.168.206.4@tcp [ 1637.171739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1637.322091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1644.446412] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:50:37 (1787748637) [ 1644.730404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.751369] LNet: 78012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1644.760480] LNetError: 78012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1644.766405] LNetError: 78012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1644.771447] LNet: Removed LNI 192.168.206.4@tcp [ 1645.807046] LNet: Removed LNI 192.168.206.4@tcp1 [ 1645.835441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.967593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1646.350332] LNet: 78053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.906804] Key type .llcrypt unregistered [ 1646.911550] Key type ._llcrypt unregistered [ 1658.754508] Key type ._llcrypt registered [ 1658.757490] Key type .llcrypt registered [ 1660.190965] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:50:53 (1787748653) [ 1660.835334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1660.847894] alg: No test for adler32 (adler32-zlib) [ 1661.763610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.321594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1665.149590] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:50:58 (1787748658) [ 1665.368937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.425955] LNet: 79152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.451657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.512316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.538292] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1666.378297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1666.402824] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1667.403725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1667.432479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp1 [ 1667.468401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp1 [ 1674.887651] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:51:07 (1787748667) [ 1676.827090] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg604-server [ 1678.740119] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:51:11 (1787748671) [ 1680.752894] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg604-server [ 1682.664106] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:51:15 (1787748675) [ 1684.613622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg604-server [ 1686.803390] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:51:19 (1787748679) [ 1688.957205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg604-server [ 1691.224142] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:51:23 (1787748683) [ 1693.794209] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg604-server [ 1695.862849] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:51:28 (1787748688) [ 1697.888590] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg604-server [ 1699.629681] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:51:32 (1787748692) [ 1702.228247] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg604-server [ 1704.226672] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:51:36 (1787748696) [ 1706.207406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg604-server [ 1708.261906] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:51:41 (1787748701) [ 1710.123379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg604-server [ 1712.066500] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:51:45 (1787748705) [ 1712.361480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.394558] LNet: 83031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.407123] LNetError: 83031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.439403] LNet: Removed LNI 192.168.206.4@tcp [ 1713.448969] LNetError: 83031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.472609] LNet: Removed LNI 192.168.206.4@tcp1 [ 1713.512924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.614406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.664191] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1714.381745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1714.476770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1714.720129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.757183] LNet: 83179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.763215] LNetError: 83179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.781601] LNet: Removed LNI 192.168.206.4@tcp [ 1714.811153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.861756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1714.881900] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1715.574107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1715.721729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.772770] LNet: Removed LNI 192.168.206.4@tcp [ 1716.809245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.918175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1716.947778] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1717.918958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1717.975786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1718.188348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.221582] LNet: 83471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.233274] LNet: 83471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1718.254784] LNetError: 83471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.263387] LNetError: 83471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1718.293959] LNet: Removed LNI 192.168.206.4@tcp [ 1718.320711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.393367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1718.427580] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1719.385601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1719.562414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.613311] LNet: Removed LNI 192.168.206.4@tcp [ 1720.657757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.720173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1720.743186] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1721.381842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1721.449818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1721.684629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.795637] LNet: Removed LNI 192.168.206.4@tcp [ 1722.839056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.901432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1722.944915] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1723.554400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1723.758744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.773426] LNet: 83909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.776956] LNet: 83909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1723.782839] LNetError: 83909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.789179] LNetError: 83909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1724.839504] LNet: Removed LNI 192.168.206.4@tcp [ 1724.889713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.005437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.023379] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1725.902705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1725.989730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1726.229506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.272415] LNet: Removed LNI 192.168.206.4@tcp [ 1727.305317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.371986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1727.409909] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1728.312779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1728.630794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.706202] LNet: Removed LNI 192.168.206.4@tcp [ 1729.756783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.843560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.887845] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1730.601647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1730.690747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1731.015701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.072795] LNet: Removed LNI 192.168.206.4@tcp [ 1732.108081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.161093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1732.181496] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1732.976781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1733.276595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.290486] LNet: 84495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.294844] LNet: 84495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1733.300431] LNetError: 84495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.310750] LNetError: 84495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1733.334771] LNet: Removed LNI 192.168.206.4@tcp [ 1733.390880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.462305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.502767] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1734.257601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1734.350780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1734.667512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.717952] LNet: Removed LNI 192.168.206.4@tcp [ 1735.733831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.796540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1735.825994] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1736.483529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 1736.760330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.841051] LNet: Removed LNI 192.168.206.4@tcp [ 1737.888926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.964133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1737.991054] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1738.595494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1738.639566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.660803] LNet: Removed LNI 192.168.206.4@tcp [ 1738.690806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.771987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1738.807731] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1739.586807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1739.673041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.718769] LNet: Removed LNI 192.168.206.4@tcp [ 1739.738093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.797414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1739.874749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.911612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.964870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1740.053697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.093459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.164342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1740.242524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.268525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.314365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1740.348597] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1741.043814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.067432] LNet: Removed LNI 192.168.206.4@tcp [ 1741.084222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.138784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1748.431229] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:52:21 (1787748741) [ 1748.583757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.622440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.665886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1748.684994] LNet: Added LNI 192.168.206.4@tcp231 [8/256/0/180] [ 1749.808789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.843468] LNet: 85946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.856940] LNet: 85946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1749.865241] LNetError: 85946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.872517] LNetError: 85946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1749.886460] LNet: Removed LNI 192.168.206.4@tcp231 [ 1749.930282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.043865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1750.087430] LNet: Added LNI 192.168.206.4@tcp231 [8/256/0/180] [ 1750.093971] LNet: 85954:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1750.565949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1750.617976] LNet: Removed LNI 192.168.206.4@tcp231 [ 1750.662840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1750.712552] LNet: Added LNI 192.168.206.4@tcp231 [8/256/0/180] [ 1751.682438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.706914] LNet: Removed LNI 192.168.206.4@tcp231 [ 1751.725650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.785649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1751.808670] LNet: Added LNI 192.168.206.4@tcp231 [8/256/0/0] [ 1751.819596] LNet: 86106:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1752.372055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1752.402812] LNet: Removed LNI 192.168.206.4@tcp231 [ 1752.432438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1752.473471] LNet: Added LNI 192.168.206.4@tcp231 [8/256/0/0] [ 1753.631544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.674145] LNet: Removed LNI 192.168.206.4@tcp231 [ 1753.700416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.792136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1753.816099] LNet: Added LNI 192.168.206.4@tcp231 [8/256/0/60] [ 1753.820789] LNet: 86258:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1754.218492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1754.248695] LNet: Removed LNI 192.168.206.4@tcp231 [ 1754.273103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1754.297837] LNet: Added LNI 192.168.206.4@tcp231 [8/256/0/60] [ 1755.414396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.460730] LNet: Removed LNI 192.168.206.4@tcp231 [ 1755.482747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.539603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1755.581888] LNet: Added LNI 192.168.206.4@tcp231 [8/256/0/180] [ 1756.114515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1756.146477] LNet: Removed LNI 192.168.206.4@tcp231 [ 1756.165533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1756.193850] LNet: Added LNI 192.168.206.4@tcp231 [8/256/0/180] [ 1764.727901] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:52:37 (1787748757) [ 1764.849143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.897667] LNet: Removed LNI 192.168.206.4@tcp231 [ 1764.934112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.046418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1765.082796] LNet: Added LNI 192.168.206.4@tcp232 [8/256/0/180] [ 1765.863487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1765.898342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1765.925221] LNet: Removed LNI 192.168.206.4@tcp232 [ 1765.939880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1765.974780] LNet: Added LNI 192.168.206.4@tcp232 [8/256/0/180] [ 1766.726248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1766.800524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1766.842530] LNet: Removed LNI 192.168.206.4@tcp232 [ 1766.872862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1766.906148] LNet: Added LNI 192.168.206.4@tcp232 [8/256/0/180] [ 1767.592438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1767.634958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1767.698777] LNet: Removed LNI 192.168.206.4@tcp232 [ 1767.714856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1767.737590] LNet: Added LNI 192.168.206.4@tcp232 [8/256/0/180] [ 1768.481830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1768.534766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1768.571778] LNet: Removed LNI 192.168.206.4@tcp232 [ 1768.589745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1775.293551] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:52:48 (1787748768) [ 1776.633875] Key type .llcrypt unregistered [ 1776.636689] Key type ._llcrypt unregistered [ 1783.227074] Key type ._llcrypt registered [ 1783.229371] Key type .llcrypt registered [ 1783.446097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1783.470825] alg: No test for adler32 (adler32-zlib) [ 1784.450304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1784.511867] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1792.675969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 1802.854771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.907362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1802.929749] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1804.479539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_interrupt -r 2 [ 1804.537357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_interrupt -r 2 [ 1804.588555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_interrupt -r 2 [ 1804.630638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_interrupt -r 2 [ 1804.666687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_interrupt -r 2 [ 1804.718995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_interrupt -r 2 [ 1804.758104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_interrupt -r 2 [ 1804.813584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_interrupt -r 2 [ 1804.865488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.4@tcp 192.168.206.104@tcp [ 1804.909373] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1804.919122] LNet: 1 local NIs in recovery (showing 1): 192.168.206.4@tcp [ 1805.241505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1805.265986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1805.294739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1805.318903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1805.342136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1805.363481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1805.398741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1805.424287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1805.461349] LNet: 88916:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1805.496977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1805.548432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1805.585402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1805.638745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1805.682789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1805.761753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1806.485330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_dropped -r 2 [ 1806.534606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_dropped -r 2 [ 1806.577532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_dropped -r 2 [ 1806.648790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_dropped -r 2 [ 1806.707906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_dropped -r 2 [ 1806.757509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_dropped -r 2 [ 1806.806300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_dropped -r 2 [ 1806.863206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_dropped -r 2 [ 1806.923627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.4@tcp 192.168.206.104@tcp [ 1806.956627] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1807.260806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1807.317762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1807.407445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1807.481244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1807.521666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1807.555618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1807.584924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1807.662886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1807.704758] LNet: 89036:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1807.738532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1807.795736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1807.856549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1807.913328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1807.962631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1808.008625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1808.638896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_aborted -r 2 [ 1808.680275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_aborted -r 2 [ 1808.729466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_aborted -r 2 [ 1808.784516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_aborted -r 2 [ 1808.831765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_aborted -r 2 [ 1808.884954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_aborted -r 2 [ 1808.939302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_aborted -r 2 [ 1809.005116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_aborted -r 2 [ 1809.081209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.4@tcp 192.168.206.104@tcp [ 1809.119147] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1809.478579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1809.511719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1809.537535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1809.557503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1809.582677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1809.622318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1809.653325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1809.703806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1809.729757] LNet: 89156:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1809.747560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1809.775321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1809.834925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1809.876185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1809.919796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1809.963239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1810.477621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_no_route -r 2 [ 1810.515636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_no_route -r 2 [ 1810.566406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_no_route -r 2 [ 1810.619599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_no_route -r 2 [ 1810.657589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_no_route -r 2 [ 1810.693913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_no_route -r 2 [ 1810.740635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_no_route -r 2 [ 1810.793960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_no_route -r 2 [ 1810.893689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.4@tcp 192.168.206.104@tcp [ 1811.270728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1811.330266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1811.370991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1811.428379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1811.470987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1811.502384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1811.548304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1811.600802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1811.650487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1811.711400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1811.772689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1811.812854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1811.869825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1811.924410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1812.500733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e local_timeout -r 2 [ 1812.535484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e local_timeout -r 2 [ 1812.568473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e local_timeout -r 2 [ 1812.622742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e local_timeout -r 2 [ 1812.660751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e local_timeout -r 2 [ 1812.708528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e local_timeout -r 2 [ 1812.750250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e local_timeout -r 2 [ 1812.805925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e local_timeout -r 2 [ 1812.856946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.4@tcp 192.168.206.104@tcp [ 1812.916124] LNet: There was an unexpected network error while writing to 192.168.206.104: rc = -22 [ 1812.921400] LNet: Skipped 1 previous similar message [ 1813.199522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1813.271855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1813.318732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1813.351983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1813.388922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1813.429442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1813.476803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1813.550065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1813.574725] LNet: 89397:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1813.584059] LNet: 89397:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1813.629358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1813.692170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1813.728724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1813.782931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1813.839416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1813.900039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1814.562483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp -m GET -e remote_dropped -r 2 [ 1814.606640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.104@tcp1 -m GET -e remote_dropped -r 2 [ 1814.672242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp -m GET -e remote_dropped -r 2 [ 1814.712488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp -d 192.168.206.4@tcp1 -m GET -e remote_dropped -r 2 [ 1814.762846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp -m GET -e remote_dropped -r 2 [ 1814.815388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.4@tcp1 -d 192.168.206.104@tcp1 -m GET -e remote_dropped -r 2 [ 1814.857883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp -m GET -e remote_dropped -r 2 [ 1814.905024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.4@tcp1 -d 192.168.206.4@tcp1 -m GET -e remote_dropped -r 2 [ 1814.946767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.4@tcp 192.168.206.104@tcp [ 1814.970939] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.104@tcp [ 1815.241246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1815.278556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1815.325104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1815.375416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1815.451127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1815.507521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1815.559462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1815.593030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1815.643405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1815.698587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1815.747466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1815.789173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1815.830620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1815.882759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1822.673882] LNet: 89595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.685466] LNetError: 89595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.711632] LNet: Removed LNI 192.168.206.4@tcp [ 1823.713728] LNetError: 89595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.740249] LNet: Removed LNI 192.168.206.4@tcp1 [ 1824.421325] Key type .llcrypt unregistered [ 1824.423324] Key type ._llcrypt unregistered [ 1843.515585] Key type ._llcrypt registered [ 1843.517944] Key type .llcrypt registered [ 1844.717565] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:53:57 (1787748837) [ 1845.316339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1845.354965] alg: No test for adler32 (adler32-zlib) [ 1846.247578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.308833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1846.324469] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1847.125967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1847.154588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.4@tcp [ 1847.174434] LNetError: 90573:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.4@tcp to recovery [ 1847.287495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.4@tcp [ 1847.406754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1854.191554] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:54:07 (1787748847) [ 1854.477407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.514724] LNet: 91163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.530703] LNetError: 91163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.566082] LNet: Removed LNI 192.168.206.4@tcp [ 1854.589534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.673992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1854.713552] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1855.436623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1855.453064] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 1856.430606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.4@tcp [ 1856.465063] LNetError: 91430:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.4@tcp to recovery [ 1856.672458] LNet: 1 local NIs in recovery (showing 1): 192.168.206.4@tcp [ 1856.673223] LNet: There was an unexpected network error while writing to 192.168.206.4: rc = -22 [ 1857.713350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.4@tcp [ 1866.645633] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:54:19 (1787748859) [ 1868.687356] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg604-server [ 1870.780298] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:54:23 (1787748863) [ 1873.350911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg604-server [ 1875.598104] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:54:28 (1787748868) [ 1877.672608] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg604-server [ 1879.692731] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:54:32 (1787748872) [ 1879.759804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1879.781941] LNet: 93043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.787343] LNetError: 93043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.799579] LNet: Removed LNI 192.168.206.4@tcp [ 1880.802070] LNetError: 93043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.813114] LNet: Removed LNI 192.168.206.4@tcp1 [ 1880.856572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.900542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1880.923903] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1881.523520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1881.581346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1881.617650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1881.729664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1881.778626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1881.868088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1887.477248] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:54:40 (1787748880) [ 1887.891503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1887.920542] LNet: 93785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.935241] LNetError: 93785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.972623] LNet: Removed LNI 192.168.206.4@tcp [ 1888.012184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.091735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1888.141980] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1895.191815] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:54:48 (1787748888) [ 1895.525868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.564825] LNet: 94488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.581344] LNetError: 94488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.601253] LNet: Removed LNI 192.168.206.4@tcp [ 1895.624314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.700237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1895.751579] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1903.066878] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:54:56 (1787748896) [ 1905.247971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1906.932276] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:54:59 (1787748899) [ 1907.496515] LNet: 95523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.506815] LNetError: 95523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.528395] LNet: Removed LNI 192.168.206.4@tcp [ 1908.100638] Key type .llcrypt unregistered [ 1908.104125] Key type ._llcrypt unregistered [ 1915.063780] Key type ._llcrypt registered [ 1915.067022] Key type .llcrypt registered [ 1915.331648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1915.356542] alg: No test for adler32 (adler32-zlib) [ 1916.422935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1916.457192] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1925.399505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 1932.003784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.104@tcp [ 1932.348176] LNet: 95950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.351981] LNetError: 95950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.363869] LNet: Removed LNI 192.168.206.4@tcp [ 1932.758552] Key type .llcrypt unregistered [ 1932.764584] Key type ._llcrypt unregistered [ 1945.852117] Key type ._llcrypt registered [ 1945.854709] Key type .llcrypt registered [ 1947.155874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1948.477870] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1950.237929] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:55:43 (1787748943) [ 1951.764565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1953.862806] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:55:46 (1787748946) [ 1956.285267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg604-server [ 1958.103933] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:55:51 (1787748951) [ 1959.884772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg604-server [ 1961.409669] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:55:54 (1787748954) [ 1963.126655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1963.157164] alg: No test for adler32 (adler32-zlib) [ 1964.219940] Lustre: Lustre: Build Version: 2.17.57_81_g37bf112 [ 1964.536287] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 1966.231186] Key type lgssc registered [ 1967.197280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1991.435634] Key type lgssc unregistered [ 1991.691420] LNet: 99127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.696907] LNetError: 99127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.708452] LNet: Removed LNI 192.168.206.4@tcp [ 1992.353436] Key type .llcrypt unregistered [ 1992.357254] Key type ._llcrypt unregistered [ 2003.947366] Key type ._llcrypt registered [ 2003.952993] Key type .llcrypt registered [ 2005.256506] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:56:38 (1787748998) [ 2005.794911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2005.805437] alg: No test for adler32 (adler32-zlib) [ 2006.696951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.757592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2006.801554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2006.857477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2006.915673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2013.590874] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:56:46 (1787749006) [ 2013.691182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2013.721732] LNet: 100538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.747838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.816748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.845864] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2014.755786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2014.774322] LNet: 100686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.778386] LNetError: 100686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.796611] LNet: Removed LNI 192.168.206.4@tcp [ 2014.840334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2014.891659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2014.930067] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2015.529750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2015.545857] LNetError: 100714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.555336] LNet: Removed LNI 192.168.206.4@tcp [ 2015.577198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2015.639338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2015.667766] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2023.187218] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:56:56 (1787749016) [ 2023.602803] LNet: 101306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.608256] LNet: 101306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2023.617829] LNetError: 101306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.632131] LNet: Removed LNI 192.168.206.4@tcp [ 2024.549506] Key type .llcrypt unregistered [ 2024.551607] Key type ._llcrypt unregistered [ 2032.761598] Key type ._llcrypt registered [ 2032.764430] Key type .llcrypt registered [ 2033.003707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2033.024394] alg: No test for adler32 (adler32-zlib) [ 2033.895547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.965869] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2033.969807] LNet: *** cfs_fail_loc=e003, val=100*** [ 2034.630660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2034.668564] LNet: *** cfs_fail_loc=e003, val=100*** [ 2034.672402] LNet: Skipped 2 previous similar messages [ 2034.680755] Ping reply received with mapping: [ 2034.684213] found metadata reply NI[192.168.206.4@tcp] [ 2034.688831] 04 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2034.695313] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2034.699410] 5a 5a 5a 5a 5a 5a 5a 5a [ 2034.844446] LNet: 101707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2034.850243] LNetError: 101707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.874387] LNet: Removed LNI 192.168.206.4@tcp [ 2035.605403] Key type .llcrypt unregistered [ 2035.607422] Key type ._llcrypt unregistered [ 2048.399690] Key type ._llcrypt registered [ 2048.401207] Key type .llcrypt registered [ 2049.474946] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:57:22 (1787749042) [ 2049.971863] Key type .llcrypt unregistered [ 2049.975096] Key type ._llcrypt unregistered [ 2056.097785] Key type ._llcrypt registered [ 2056.104456] Key type .llcrypt registered [ 2056.383549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2056.425320] alg: No test for adler32 (adler32-zlib) [ 2057.709892] LNetError: 102765:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2058.327666] Key type .llcrypt unregistered [ 2058.336862] Key type ._llcrypt unregistered [ 2064.170851] Key type ._llcrypt registered [ 2064.172487] Key type .llcrypt registered [ 2065.670769] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:57:38 (1787749058) [ 2066.560422] Key type .llcrypt unregistered [ 2066.564966] Key type ._llcrypt unregistered [ 2075.550856] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2075.556973] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2078.269515] Key type ._llcrypt registered [ 2078.273099] Key type .llcrypt registered [ 2078.460946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2078.475534] alg: No test for adler32 (adler32-zlib) [ 2079.427204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.508933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2090.583853] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:58:03 (1787749083) [ 2090.798960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2090.833900] LNet: 104548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.854640] LNetError: 104548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2090.917394] LNet: Removed LNI 10.1.2.3@tcp [ 2090.950540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.995078] Key type .llcrypt unregistered [ 2092.000123] Key type ._llcrypt unregistered [ 2099.428926] Key type ._llcrypt registered [ 2099.432591] Key type .llcrypt registered [ 2099.637453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2099.658774] alg: No test for adler32 (adler32-zlib) [ 2100.619387] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2100.697382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2100.878852] LNet: 104846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.892919] LNetError: 104846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.921943] LNet: Removed LNI 192.168.206.4@tcp [ 2101.765363] Key type .llcrypt unregistered [ 2101.767969] Key type ._llcrypt unregistered [ 2114.890909] Key type ._llcrypt registered [ 2114.894972] Key type .llcrypt registered [ 2116.263786] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:58:29 (1787749109) [ 2117.292583] Key type .llcrypt unregistered [ 2117.294781] Key type ._llcrypt unregistered [ 2124.207248] Key type ._llcrypt registered [ 2124.211120] Key type .llcrypt registered [ 2124.471979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2124.509543] alg: No test for adler32 (adler32-zlib) [ 2132.216126] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:58:45 (1787749125) [ 2132.366392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2132.420830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.490543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2132.520722] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2133.163750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2133.190794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2133.216119] LNet: Removed LNI 192.168.206.4@tcp [ 2133.531638] LNet: 106603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.969548] Key type .llcrypt unregistered [ 2133.973581] Key type ._llcrypt unregistered [ 2146.543503] Key type ._llcrypt registered [ 2146.548118] Key type .llcrypt registered [ 2147.882083] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:59:00 (1787749140) [ 2148.331976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.352104] alg: No test for adler32 (adler32-zlib) [ 2149.363960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.413420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2149.438287] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2150.227325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2150.301462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.4@tcp [ 2150.329586] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2150.335766] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.339531] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 0 [ 2150.343896] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2150.348711] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2150.351905] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.355741] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 0 [ 2150.362778] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2150.365777] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2150.371996] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.378647] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 2200 [ 2150.383777] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2150.386222] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2150.390394] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.393805] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 2200 [ 2150.399365] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2158.214552] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:59:10 (1787749150) [ 2158.751667] LNet: 108143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.773200] LNetError: 108143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.806944] LNet: Removed LNI 192.168.206.4@tcp [ 2159.758845] Key type .llcrypt unregistered [ 2159.760867] Key type ._llcrypt unregistered [ 2167.650400] Key type ._llcrypt registered [ 2167.653767] Key type .llcrypt registered [ 2167.910716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2167.935390] alg: No test for adler32 (adler32-zlib) [ 2169.002409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2169.083583] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2178.017868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 2186.839295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.889614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2186.926899] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 2188.010485] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2188.252535] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2188.260425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2188.272311] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2190.738487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2190.779708] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2203.110407] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:59:55 (1787749195) [ 2203.400782] LNet: 109649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.409815] LNetError: 109649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.416585] LNet: Removed LNI 192.168.206.4@tcp [ 2204.447538] LNet: Removed LNI 192.168.206.4@tcp1 [ 2204.450135] LNetError: 109649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2204.462570] LNetError: 109649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2204.485989] LNet: Removed LNI 10.1.2.3@tcp99 [ 2205.304566] Key type .llcrypt unregistered [ 2205.309830] Key type ._llcrypt unregistered [ 2213.206246] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2213.211149] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2215.717340] Key type ._llcrypt registered [ 2215.718912] Key type .llcrypt registered [ 2215.914670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2215.936463] alg: No test for adler32 (adler32-zlib) [ 2216.883312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.967747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2217.005841] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2217.931434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2217.971902] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2220.543064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2220.613926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2220.701393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2220.744708] LNetError: 110400:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2220.792869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2220.877884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2221.442129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2221.561139] LNet: 110420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.569450] LNetError: 110420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2221.580212] LNet: Removed LNI 10.1.2.3@tcp [ 2222.631538] LNet: Removed LNI 192.168.206.4@tcp [ 2223.514682] Key type .llcrypt unregistered [ 2223.517349] Key type ._llcrypt unregistered [ 2238.313587] Key type ._llcrypt registered [ 2238.319372] Key type .llcrypt registered [ 2239.728618] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:00:32 (1787749232) [ 2240.410536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2240.427517] alg: No test for adler32 (adler32-zlib) [ 2241.398594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.487800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.524752] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2242.420890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2242.514945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg604-client@tcp [ 2250.670958] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:00:43 (1787749243) [ 2250.848138] LNet: 111966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.862582] LNetError: 111966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.885704] LNet: Removed LNI 192.168.206.4@tcp [ 2253.010596] Key type .llcrypt unregistered [ 2253.014444] Key type ._llcrypt unregistered [ 2260.330385] Key type ._llcrypt registered [ 2260.332580] Key type .llcrypt registered [ 2260.660851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2260.679785] alg: No test for adler32 (adler32-zlib) [ 2261.565373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2261.602704] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2261.669076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2270.023747] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:01:02 (1787749262) [ 2272.106517] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2274.017586] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:01:06 (1787749266) [ 2275.985181] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2278.351677] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:01:10 (1787749270) [ 2278.746338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.806359] LNet: 113413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.821707] LNetError: 113413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.845890] LNet: Removed LNI 192.168.206.4@tcp [ 2278.881465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2278.976580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.015523] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2279.952758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2280.050875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2287.833881] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:01:20 (1787749280) [ 2288.148312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2288.184989] LNet: 114123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.197732] LNetError: 114123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.229361] LNet: Removed LNI 192.168.206.4@tcp [ 2288.272550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.353686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2288.420266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2289.172354] Key type .llcrypt unregistered [ 2289.174899] Key type ._llcrypt unregistered [ 2301.820528] Key type ._llcrypt registered [ 2301.822649] Key type .llcrypt registered [ 2303.325761] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:01:36 (1787749296) [ 2303.819826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2303.831145] alg: No test for adler32 (adler32-zlib) [ 2304.730970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.796149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.820846] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2305.507852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2305.561720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2312.786785] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:01:45 (1787749305) [ 2313.331180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2313.369127] LNet: 115684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2313.384986] LNetError: 115684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2314.411967] LNet: Removed LNI 192.168.206.4@tcp [ 2314.469813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.571635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2314.633534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2321.975469] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:01:54 (1787749314) [ 2322.294471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.342338] LNet: 116266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.366731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.473840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2322.979757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2323.022732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2323.123569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2323.718662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2323.749671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2323.829376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2323.890647] LNet: 116299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.934766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.023670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2324.561123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2324.617616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2324.665724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2325.135578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2325.170910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2333.110160] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:02:05 (1787749325) [ 2333.378249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.405942] LNet: 116901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.425531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.464719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2333.482212] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2334.332363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2334.357956] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2335.298172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2335.376268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2335.582831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.606716] LNetError: 117180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.611887] LNet: Removed LNI 192.168.206.4@tcp [ 2335.620512] LNet: Removed LNI 192.168.206.4@tcp2 [ 2335.645871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.713937] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2335.719910] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2335.730265] LNet: 117186:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2336.378968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2336.556700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.631153] LNet: Removed LNI 192.168.206.4@tcp [ 2337.633508] LNetError: 117223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.648699] LNetError: 117223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2337.668790] LNet: Removed LNI 192.168.206.4@tcp2 [ 2337.711699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.812140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2337.846900] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2338.801610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2338.856056] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2339.910197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2339.959270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2340.053680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2340.384019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.421749] LNet: 117508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.433981] LNet: 117508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2340.446530] LNetError: 117508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.463060] LNet: Removed LNI 192.168.206.4@tcp [ 2341.480724] LNet: Removed LNI 192.168.206.4@tcp2 [ 2341.523546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.644270] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2341.654499] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2341.659866] LNet: 117514:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2342.319632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2342.613827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.649627] LNet: Removed LNI 192.168.206.4@tcp [ 2343.718606] LNet: Removed LNI 192.168.206.4@tcp2 [ 2343.768976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.862522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2343.944596] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2344.741367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2344.758916] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2345.820193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2345.889511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2346.062837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.094329] LNetError: 117831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.104753] LNetError: 117831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2346.113894] LNet: Removed LNI 192.168.206.4@tcp [ 2346.131439] LNet: Removed LNI 192.168.206.4@tcp2 [ 2346.154952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.222505] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2346.229959] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2346.236398] LNet: 117837:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2346.788460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2346.969729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.063323] LNet: Removed LNI 192.168.206.4@tcp [ 2348.096113] LNet: Removed LNI 192.168.206.4@tcp2 [ 2348.128081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.216896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2348.247525] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2348.934815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2348.959228] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2349.801855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2349.836327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2349.890637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2350.135691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2350.150288] LNet: 118158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2350.152903] LNet: 118158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2351.199649] LNet: Removed LNI 192.168.206.4@tcp [ 2352.232407] LNet: Removed LNI 192.168.206.4@tcp2 [ 2352.277448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2352.353355] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2352.359810] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2352.366076] LNet: 118165:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2352.776282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2352.954916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.973826] LNet: Removed LNI 192.168.206.4@tcp [ 2354.024759] LNet: Removed LNI 192.168.206.4@tcp2 [ 2354.059815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.111668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2354.166665] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2355.159277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2355.187923] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2355.970737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2356.012953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2356.055456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2356.397301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2356.413602] LNetError: 118498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.416516] LNetError: 118498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2357.471287] LNet: Removed LNI 192.168.206.4@tcp [ 2357.492353] LNet: Removed LNI 192.168.206.4@tcp2 [ 2357.514493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.585941] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2357.593590] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2357.596981] LNet: 118504:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2357.945284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2358.164232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.199182] LNet: Removed LNI 192.168.206.4@tcp [ 2359.218370] LNet: Removed LNI 192.168.206.4@tcp2 [ 2359.259289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.350220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2359.386206] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2360.385789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2360.426357] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2361.392178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2361.492824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2361.554279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2361.647389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2362.027081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2363.103297] LNet: Removed LNI 192.168.206.4@tcp [ 2364.135043] LNet: Removed LNI 192.168.206.4@tcp2 [ 2364.165987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.232703] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2364.242642] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2364.846671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2375.422848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2377.012477] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:02:50 (1787749370) [ 2377.211485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2377.259463] LNet: 119622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2377.265345] LNet: 119622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2377.296951] LNetError: 119622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.307698] LNetError: 119622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2377.330294] LNet: Removed LNI 192.168.206.4@tcp [ 2378.354858] LNet: Removed LNI 192.168.206.4@tcp2 [ 2378.397340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.767786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2378.799352] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2379.673677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2379.717534] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2380.755974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.791701] LNet: Removed LNI 10.1.2.3@tcp [ 2380.815331] LNet: Removed LNI 192.168.206.4@tcp [ 2380.833314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.929395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2380.973286] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2380.978141] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2381.396374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2381.412853] LNet: Removed LNI 192.168.206.4@tcp [ 2381.420919] LNet: Removed LNI 10.1.2.3@tcp [ 2388.728968] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 09:03:01 (1787749381) [ 2389.165197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2389.209179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2389.301764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2389.325492] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2389.977061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2390.011602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2390.050798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2390.092907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2390.140085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2390.206166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2390.269145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2390.344509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2390.390348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2390.424212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2390.481554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.4@tcp [ 2390.672861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2397.676049] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:03:10 (1787749390) [ 2397.985370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2399.101874] LNet: Removed LNI 192.168.206.4@tcp [ 2399.130549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.326901] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2399.510082] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2399.529159] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2399.535553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2401.986959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2402.019582] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2402.927744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2402.980059] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2404.778975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2404.820453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2404.895068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.4@tcp -m PUT -e local_timeout -r 1 [ 2404.962778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2405.027637] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2405.031499] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2405.216537] LNetError: 121301:0:(lib-move.c:4801:lnet_parse()) 192.168.206.4@tcp, src 192.168.206.4@tcp: Dropping PUT (error -108 looking up sender) [ 2406.244764] LNet: Removed LNI 10.1.2.3@tcp [ 2407.293254] LNet: Removed LNI 192.168.206.4@tcp [ 2407.859613] Key type .llcrypt unregistered [ 2407.872869] Key type ._llcrypt unregistered [ 2421.889783] Key type ._llcrypt registered [ 2421.892386] Key type .llcrypt registered [ 2423.384161] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:03:36 (1787749416) [ 2423.876513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2423.903713] alg: No test for adler32 (adler32-zlib) [ 2424.825930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2425.158281] LNet: 122617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.866688] Key type .llcrypt unregistered [ 2425.873737] Key type ._llcrypt unregistered [ 2433.407902] Key type ._llcrypt registered [ 2433.410095] Key type .llcrypt registered [ 2433.643190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2433.665263] alg: No test for adler32 (adler32-zlib) [ 2434.675973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2434.738730] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2443.296845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 2443.556601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2444.449847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.104@tcp [ 2444.814902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2444.863700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2444.895718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2444.934597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2444.983137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2445.026686] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.104@tcp [ 2445.033542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2445.069333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2445.118321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2445.145216] LNet: 123071:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2445.166238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2445.208762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2445.271100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2445.313576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2445.387087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2445.423155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2446.135048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2446.384656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2446.429898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2446.486760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2446.543689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2446.602857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2446.640380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2446.682305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2446.714936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2446.741032] LNet: 123188:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2446.770127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2446.806219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2446.849878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2446.919818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2446.965650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2447.009815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2447.766714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.104@tcp [ 2448.057457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2448.099913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2448.131909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2448.168376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2448.198733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2448.228881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2448.266301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2448.299722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2448.323425] LNet: 123305:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2448.344023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2448.387651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2448.425980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2448.463813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2448.498844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2448.551223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2449.447477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2449.685600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2449.709593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2449.737423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2449.771888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2449.798745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2449.831479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2449.853529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2449.889612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2449.920791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2449.945573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2449.986689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2450.018389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2450.047803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2450.094850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2450.619589] LNet: 123472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.642609] LNetError: 123472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2450.661882] LNet: Removed LNI 192.168.206.4@tcp [ 2451.269827] Key type .llcrypt unregistered [ 2451.274531] Key type ._llcrypt unregistered [ 2470.640254] Key type ._llcrypt registered [ 2470.652750] Key type .llcrypt registered [ 2471.882977] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:04:24 (1787749464) [ 2472.588112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2472.620342] alg: No test for adler32 (adler32-zlib) [ 2473.477685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2473.785643] LNet: 124359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.397811] Key type .llcrypt unregistered [ 2474.400600] Key type ._llcrypt unregistered [ 2482.258757] Key type ._llcrypt registered [ 2482.261503] Key type .llcrypt registered [ 2482.578089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2482.615759] alg: No test for adler32 (adler32-zlib) [ 2483.616959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2483.672819] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2493.049881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.104@tcp [ 2504.578358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2504.629816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2504.655186] LNet: Added LNI 192.168.206.4@tcp1 [8/256/0/180] [ 2505.470478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2505.868418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.104@tcp [ 2506.069630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2506.123356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2506.149756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2506.182575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2506.207959] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.104@tcp [ 2506.226740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2506.252322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2506.275030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2506.298312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2506.319266] LNet: 125016:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2506.335341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2506.368275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2506.396896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2506.435785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2506.463837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2506.495020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2507.163742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2507.423882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2507.449542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2507.485783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2507.526959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2507.553193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2507.582689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2507.613193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2507.634685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2507.663582] LNet: 125133:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2507.683781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2507.727331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2507.747530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2507.767665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2507.788813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2507.805877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2508.333456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.104@tcp [ 2508.514258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2508.558207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2508.586532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2508.640604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2508.679286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2508.702715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2508.728759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2508.774888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2508.795988] LNet: 125250:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2508.806389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2508.845286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2508.890493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2508.920183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2508.950170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2508.986410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2509.535709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2509.749464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2509.786297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2509.813440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2509.841805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2509.883571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2509.946559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2510.001253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2510.049785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2510.090925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2510.126959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2510.163594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2510.205845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2510.275873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2510.345209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2515.005516] LNet: 125452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2515.025385] LNetError: 125452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2515.035803] LNet: Removed LNI 192.168.206.4@tcp [ 2515.066847] LNet: Removed LNI 192.168.206.4@tcp1 [ 2515.625728] Key type .llcrypt unregistered [ 2515.630385] Key type ._llcrypt unregistered [ 2533.829219] Key type ._llcrypt registered [ 2533.831715] Key type .llcrypt registered [ 2535.231923] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 09:05:28 (1787749528) [ 2535.751756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2535.761646] alg: No test for adler32 (adler32-zlib) [ 2536.761230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2536.812484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2536.843001] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2537.446659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2537.465962] LNet: Added LNI 192.168.206.4@tcp2 [8/256/0/180] [ 2538.270910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.4@tcp [ 2538.371989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2539.424197] LNet: Removed LNI 192.168.206.4@tcp [ 2547.570243] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 09:05:40 (1787749540) [ 2549.491505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2551.553448] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:05:44 (1787749544) [ 2559.904778] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:05:52 (1787749552) [ 2568.278294] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:06:01 (1787749561) [ 2576.810770] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:06:09 (1787749569) [ 2586.657781] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:06:18 (1787749578) [ 2596.514454] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:06:29 (1787749589) [ 2605.664970] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:06:38 (1787749598) [ 2615.577869] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:06:48 (1787749608) [ 2624.484585] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:06:57 (1787749617) [ 2632.753643] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:07:05 (1787749625) [ 2641.646038] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:07:13 (1787749633) [ 2650.085950] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:07:23 (1787749643) [ 2658.660494] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:07:31 (1787749651) [ 2666.548631] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:07:39 (1787749659) [ 2675.774581] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:07:48 (1787749668) [ 2683.794270] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:07:56 (1787749676) [ 2691.626664] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:08:04 (1787749684) [ 2699.503587] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:08:12 (1787749692) [ 2699.708084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2699.756365] LNet: 137149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2699.762492] LNetError: 137149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.840636] LNet: Removed LNI 192.168.206.4@tcp2 [ 2700.924303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2701.112526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2701.142163] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2702.352568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2702.372976] LNet: 137291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2702.384737] LNetError: 137291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2702.407689] LNet: Removed LNI 192.168.206.4@tcp [ 2702.448751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2702.519919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2702.540044] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2710.449279] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:08:23 (1787749703) [ 2718.677659] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:08:31 (1787749711) [ 2726.935838] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:08:39 (1787749719) [ 2735.220668] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:08:47 (1787749727) [ 2742.281913] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:08:55 (1787749735) [ 2749.753868] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:09:02 (1787749742) [ 2758.909262] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:09:11 (1787749751) [ 2768.230415] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:09:21 (1787749761) [ 2776.557600] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:09:29 (1787749769) [ 2785.009131] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:09:38 (1787749778) [ 2791.773548] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:09:44 (1787749784) [ 2798.687744] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:09:51 (1787749791) [ 2806.809400] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2547 sec ========== 09:09:59 (1787749799) [ 2807.202322] LNet: 144760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2807.215400] LNetError: 144760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2807.238512] LNet: Removed LNI 192.168.206.4@tcp [ 2808.135628] Key type .llcrypt unregistered [ 2808.138897] Key type ._llcrypt unregistered [ 2822.861986] Key type ._llcrypt registered [ 2822.866326] Key type .llcrypt registered [ 2823.717151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2823.754776] alg: No test for adler32 (adler32-zlib) [ 2824.844817] Lustre: Lustre: Build Version: 2.17.57_81_g37bf112 [ 2825.287571] LNet: Added LNI 192.168.206.4@tcp [8/256/0/180] [ 2827.015221] Key type lgssc registered [ 2828.367967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2902.286896] Lustre: Mounted lustre-client - version 2.17.57_81_g37bf112 [ 2908.113258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2917.011944] Lustre: Unmounted lustre-client [ 2942.360780] Key type lgssc unregistered [ 2942.658865] LNet: 147717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2942.666398] LNetError: 147717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2942.684936] LNet: Removed LNI 192.168.206.4@tcp [ 2943.394149] Key type .llcrypt unregistered [ 2943.397957] Key type ._llcrypt unregistered