[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 400538634 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003130] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.007948] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011040] Yama: becoming mindful. [ 0.012024] SELinux: Initializing. [ 0.013056] *** VALIDATE selinux *** [ 0.020649] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024713] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028030] *** VALIDATE tmpfs *** [ 0.029357] *** VALIDATE proc *** [ 0.030153] *** VALIDATE cgroup *** [ 0.031004] *** VALIDATE cgroup2 *** [ 0.032216] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035028] Spectre V2 : User space: Vulnerable [ 0.036004] Speculative Store Bypass: Vulnerable [ 0.039271] debug: unmapping init [mem 0xffffffff85e59000-0xffffffff85e60fff] [ 0.042145] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043682] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044026] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049010] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051274] rcu: Hierarchical SRCU implementation. [ 0.053263] smp: Bringing up secondary CPUs ... [ 0.054539] x86: Booting SMP configuration: [ 0.055028] .... node #0, CPUs: #1 #2 #3 [ 0.058256] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.210000] node 0 deferred pages initialised in 146ms [ 0.214081] devtmpfs: initialized [ 0.215666] x86/mm: Memory block size: 128MB [ 0.219072] gcov: version magic: 0x41383552 [ 0.221330] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.225218] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.227415] pinctrl core: initialized pinctrl subsystem [ 0.229258] [ 0.229763] ************************************************************* [ 0.231011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.233014] ** ** [ 0.235011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.237013] ** ** [ 0.239013] ** This means that this kernel is built to expose internal ** [ 0.242015] ** IOMMU data structures, which may compromise security on ** [ 0.244013] ** your system. ** [ 0.245011] ** ** [ 0.247012] ** If you see this message and you are not debugging the ** [ 0.249013] ** kernel, report this immediately to your vendor! ** [ 0.251011] ** ** [ 0.253013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.255023] ************************************************************* [ 0.257726] NET: Registered protocol family 16 [ 0.259464] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.262068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.264063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.267201] cpuidle: using governor menu [ 0.268661] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.271507] PCI: Using configuration type 1 for base access [ 0.272152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.282128] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.283028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.284326] cryptd: max_cpu_qlen set to 1000 [ 0.286096] ACPI: Added _OSI(Module Device) [ 0.287008] ACPI: Added _OSI(Processor Device) [ 0.288008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.289026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.293649] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.300012] ACPI: Interpreter enabled [ 0.301061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.302012] ACPI: Using IOAPIC for interrupt routing [ 0.303138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.306654] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.316609] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.319042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.322027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.326137] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.332812] acpiphp: Slot [2] registered [ 0.334117] acpiphp: Slot [5] registered [ 0.334990] acpiphp: Slot [6] registered [ 0.336119] acpiphp: Slot [3] registered [ 0.338089] acpiphp: Slot [4] registered [ 0.339360] acpiphp: Slot [7] registered [ 0.341102] acpiphp: Slot [8] registered [ 0.342143] acpiphp: Slot [9] registered [ 0.344114] acpiphp: Slot [10] registered [ 0.345104] acpiphp: Slot [11] registered [ 0.346064] acpiphp: Slot [12] registered [ 0.347028] acpiphp: Slot [13] registered [ 0.349150] acpiphp: Slot [14] registered [ 0.351005] acpiphp: Slot [15] registered [ 0.352069] acpiphp: Slot [16] registered [ 0.353081] acpiphp: Slot [17] registered [ 0.354091] acpiphp: Slot [18] registered [ 0.355042] acpiphp: Slot [19] registered [ 0.356073] acpiphp: Slot [20] registered [ 0.358131] acpiphp: Slot [21] registered [ 0.359059] acpiphp: Slot [22] registered [ 0.360060] acpiphp: Slot [23] registered [ 0.361125] acpiphp: Slot [24] registered [ 0.362082] acpiphp: Slot [25] registered [ 0.363091] acpiphp: Slot [26] registered [ 0.365085] acpiphp: Slot [27] registered [ 0.366119] acpiphp: Slot [28] registered [ 0.368082] acpiphp: Slot [29] registered [ 0.369093] acpiphp: Slot [30] registered [ 0.370131] acpiphp: Slot [31] registered [ 0.372060] PCI host bridge to bus 0000:00 [ 0.373017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.375025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.377027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.380025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.382035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.384029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.386188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.388975] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.392351] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.399016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.403036] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.410021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.413017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.416017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.420676] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.423813] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.427077] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.431105] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.436017] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.445602] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.450017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.454929] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.464856] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.468011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.479014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.491000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.499018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.508022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.539247] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.541434] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.544061] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.546475] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.548250] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.557155] iommu: Default domain type: Passthrough [ 0.561742] SCSI subsystem initialized [ 0.564496] ACPI: bus type USB registered [ 0.567164] usbcore: registered new interface driver usbfs [ 0.570111] usbcore: registered new interface driver hub [ 0.572124] usbcore: registered new device driver usb [ 0.575370] pps_core: LinuxPPS API ver. 1 registered [ 0.577012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.580086] PTP clock support registered [ 0.582249] EDAC MC: Ver: 3.0.0 [ 0.584442] PCI: Using ACPI for IRQ routing [ 0.588238] NetLabel: Initializing [ 0.590014] NetLabel: domain hash size = 128 [ 0.592016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.595103] NetLabel: unlabeled traffic allowed by default [ 0.598187] vgaarb: loaded [ 0.599686] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.603020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.612000] clocksource: Switched to clocksource kvm-clock [ 0.750699] VFS: Disk quotas dquot_6.6.0 [ 0.752401] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.755315] *** VALIDATE ramfs *** [ 0.756849] *** VALIDATE hugetlbfs *** [ 0.758636] pnp: PnP ACPI init [ 0.761933] pnp: PnP ACPI: found 6 devices [ 0.778373] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.781783] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.784181] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.786115] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.788405] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.790660] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.793481] NET: Registered protocol family 2 [ 0.797744] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.805255] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.808930] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.814707] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.817927] TCP: Hash tables configured (established 65536 bind 65536) [ 0.820964] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.824274] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.827558] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.830578] NET: Registered protocol family 1 [ 0.833079] RPC: Registered named UNIX socket transport module. [ 0.834686] RPC: Registered udp transport module. [ 0.836586] RPC: Registered tcp transport module. [ 0.837911] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.839600] NET: Registered protocol family 44 [ 0.841509] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.843508] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.845598] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.847816] PCI: CLS 0 bytes, default 64 [ 0.849711] Unpacking initramfs... [ 2.314820] debug: unmapping init [mem 0xffff9387fcc64000-0xffff9387fffcffff] [ 2.321128] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.323456] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.327906] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.851178] Initialise system trusted keyrings [ 2.852805] Key type blacklist registered [ 2.856400] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.872448] zbud: loaded [ 2.876129] *** VALIDATE nfs *** [ 2.877255] *** VALIDATE nfs4 *** [ 2.878853] pstore: using deflate compression [ 2.882921] Platform Keyring initialized [ 2.990740] NET: Registered protocol family 38 [ 2.992933] Key type asymmetric registered [ 2.994585] Asymmetric key parser 'x509' registered [ 2.996695] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.999648] io scheduler mq-deadline registered [ 3.001218] io scheduler kyber registered [ 3.002807] io scheduler bfq registered [ 3.004894] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.007908] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.010530] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.012939] ACPI: Power Button [PWRF] [ 3.018013] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.024837] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.043495] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.073532] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.102659] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.109968] Non-volatile memory driver v1.3 [ 3.111416] Linux agpgart interface v0.103 [ 3.141199] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.143609] vda: detected capacity change from 0 to 74825728 [ 3.159603] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.163049] vdb: detected capacity change from 0 to 1073741824 [ 3.171525] libphy: Fixed MDIO Bus: probed [ 3.176374] usbcore: registered new interface driver usbserial_generic [ 3.178403] usbserial: USB Serial support registered for generic [ 3.180363] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.183837] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.185472] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.187729] mousedev: PS/2 mouse device common for all mice [ 3.191779] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.193376] rtc_cmos 00:05: RTC can wake from S4 [ 3.200541] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.205112] rtc_cmos 00:05: registered as rtc0 [ 3.209176] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.209907] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.213134] intel_pstate: CPU model not supported [ 3.215307] hid: raw HID events driver (C) Jiri Kosina [ 3.222417] usbcore: registered new interface driver usbhid [ 3.224622] usbhid: USB HID core driver [ 3.225856] drop_monitor: Initializing network drop monitor service [ 3.228071] Initializing XFRM netlink socket [ 3.230703] NET: Registered protocol family 10 [ 3.234233] Segment Routing with IPv6 [ 3.235589] NET: Registered protocol family 17 [ 3.237422] mpls_gso: MPLS GSO support [ 3.243434] RAS: Correctable Errors collector initialized. [ 3.245245] AVX version of gcm_enc/dec engaged. [ 3.246871] AES CTR mode by8 optimization enabled [ 3.318268] sched_clock: Marking stable (3318188892, 0)->(4158417577, -840228685) [ 3.322227] registered taskstats version 1 [ 3.324807] Loading compiled-in X.509 certificates [ 3.327240] zswap: loaded using pool lzo/zbud [ 3.356768] Key type big_key registered [ 3.369485] Key type encrypted registered [ 3.371595] ima: No TPM chip found, activating TPM-bypass! [ 3.375347] ima: Allocated hash algorithm: sha1 [ 3.377990] ima: No architecture policies found [ 3.380975] evm: Initialising EVM extended attributes: [ 3.383769] evm: security.selinux [ 3.385952] evm: security.ima [ 3.387638] evm: security.capability [ 3.389264] evm: HMAC attrs: 0x1 [ 3.392244] rtc_cmos 00:05: setting system clock to 2026-09-01 14:03:55 UTC (1788271435) [ 3.397869] debug: unmapping init [mem 0xffffffff86e03000-0xffffffff86ffffff] [ 3.400296] debug: unmapping init [mem 0xffffffff85b82000-0xffffffff85e58fff] [ 3.408117] Write protecting the kernel read-only data: 28672k [ 3.411034] debug: unmapping init [mem 0xffffffff84203000-0xffffffff843fffff] [ 3.412994] debug: unmapping init [mem 0xffffffff84b14000-0xffffffff84bfffff] [ 3.444722] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.453325] systemd[1]: Detected virtualization kvm. [ 3.455582] systemd[1]: Detected architecture x86-64. [ 3.458092] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.487387] systemd[1]: No hostname configured. [ 3.489286] systemd[1]: Set hostname to . [ 3.491611] random: systemd: uninitialized urandom read (16 bytes read) [ 3.494532] systemd[1]: Initializing machine ID from random generator. [ 3.545710] random: ln: uninitialized urandom read (6 bytes read) [ 3.669099] random: systemd: uninitialized urandom read (16 bytes read) [ 3.672468] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.676585] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.682793] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.495198] device-mapper: uevent: version 1.0.3 [ 4.497267] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.677764] virtio_net virtio0 ens2: renamed from eth0 [ 5.713181] random: fast init done [ 6.004166] scsi host0: ata_piix [ 6.051633] scsi host1: ata_piix [ 6.053074] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.055365] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 14.488827] random: crng init done [ 14.490666] random: 7 urandom warning(s) missed due to ratelimiting [ 15.094376] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.988164] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.707899] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.137636] SELinux: Disabled at runtime. [ 18.361227] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.379541] systemd[1]: Detected virtualization kvm. [ 18.384647] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.114102] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.177049] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.196421] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.202693] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.207966] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.217364] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.227544] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ 19.376014] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 20.279084] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.240052] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.401306] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.525949] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.921364] EDAC sbridge: Ver: 1.1.2 [ 24.542059] Key type dns_resolver registered [ 24.933429] NFS: Registering the id_resolver key type [ 24.935942] Key type id_resolver registered [ 24.938059] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg348-client login: [ 60.613354] libcfs: loading out-of-tree module taints kernel. [ 60.686817] Key type ._llcrypt registered [ 60.688546] Key type .llcrypt registered [ 60.972854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 60.979995] alg: No test for adler32 (adler32-zlib) [ 62.156631] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 62.629908] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 64.311738] Key type lgssc registered [ 65.070231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 158.021116] hrtimer: interrupt took 10107221 ns [ 174.169586] Lustre: Mounted lustre-client - version 2.17.57_64_g65721f0 [ 178.495435] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 192.105700] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing check_logdir /tmp/testlogs/ [ 195.832654] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing yml_node [ 198.959286] Lustre: DEBUG MARKER: Client: 2.17.57.64 [ 199.649747] Lustre: lustre-OST0000-osc-ffff938848790000: disconnect after 23s idle [ 200.895820] Lustre: DEBUG MARKER: MDS: 2.17.57.64 [ 202.690559] Lustre: DEBUG MARKER: OSS: 2.17.57.64 [ 203.734775] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 1 10:07:14 EDT 2026 [ 215.848374] Lustre: Unmounted lustre-client [ 234.955704] Key type lgssc unregistered [ 235.140583] LNet: 8425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 235.150382] LNetError: 8425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 235.168826] LNet: Removed LNI 192.168.203.48@tcp [ 235.641386] Key type .llcrypt unregistered [ 235.644685] Key type ._llcrypt unregistered [ 256.412919] Key type ._llcrypt registered [ 256.418887] Key type .llcrypt registered [ 256.566851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 256.584252] alg: No test for adler32 (adler32-zlib) [ 257.451460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 257.520439] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 257.551354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 260.778818] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 261.097746] LNet: 9312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 261.105841] LNetError: 9312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 261.138891] LNet: Removed LNI 192.168.203.48@tcp [ 261.878528] Key type .llcrypt unregistered [ 261.881338] Key type ._llcrypt unregistered [ 266.968109] Key type ._llcrypt registered [ 266.973877] Key type .llcrypt registered [ 267.818847] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:08:18 (1788271698) [ 268.106671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 268.123143] alg: No test for adler32 (adler32-zlib) [ 268.990565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.045385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 269.066106] LNet: 9726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 269.087239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.128833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 269.149553] LNet: 9733:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 274.540874] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:08:25 (1788271705) [ 274.698608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.716269] LNet: 10310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 274.730888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.798416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 279.807872] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:08:30 (1788271710) [ 279.910370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.930114] LNet: 10892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.958454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.014900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 285.204792] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:08:36 (1788271716) [ 285.287606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.323690] LNet: 11472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.373226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.492515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 291.108747] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:08:41 (1788271721) [ 291.349557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 291.371836] LNet: 12063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 291.416531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.481927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 291.979863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 296.864486] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:08:48 (1788271728) [ 297.041945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.078914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.158483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 297.411763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 297.701387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 303.063858] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:08:54 (1788271734) [ 303.126236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.142334] LNet: 13246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.148173] LNet: 13246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 303.163423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.227384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 309.391663] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:09:00 (1788271740) [ 309.701145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.762361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.833939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 309.870916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 310.316445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 310.345434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 310.618420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 310.677948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 311.267331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 311.339566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 311.740927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 311.775523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 312.128796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 312.162286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 312.738179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 312.797376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 313.215582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 313.306026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 313.918989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 313.971539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 314.576916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 314.617500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 315.170747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 315.241839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 315.825362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 315.890628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 316.470245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 316.561562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 323.482042] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:09:14 (1788271754) [ 323.626180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.655431] LNet: 14515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.666796] LNet: 14515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 323.703250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.787949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 323.832200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 329.827742] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:09:20 (1788271760) [ 329.971758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.018796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.104555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 330.148897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 336.136096] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:09:26 (1788271766) [ 336.228623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.272575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.375020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 336.428317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 342.520405] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:09:33 (1788271773) [ 342.707568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.764275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.935116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 342.973187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 350.522934] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:09:40 (1788271780) [ 350.600614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.662297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.737667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 350.811287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 359.552829] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:09:49 (1788271789) [ 359.676204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.723172] LNet: 17442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.743837] LNet: 17442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 359.805073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.987249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 360.104740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 366.150352] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:09:57 (1788271797) [ 366.321317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.352512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.407912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 366.791687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 372.932484] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:10:03 (1788271803) [ 373.223581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.294168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.562659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 373.858311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 381.264978] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:10:11 (1788271811) [ 381.665401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.735515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.015367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 382.335783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 388.725371] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:10:19 (1788271819) [ 389.086504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.150431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.805867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 389.985407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 396.194421] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:10:27 (1788271827) [ 396.320790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.390095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.497020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 396.552438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 401.956551] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:10:32 (1788271832) [ 402.256232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.317928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.400759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 402.458932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 408.046080] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:10:39 (1788271839) [ 408.177267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.218980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.274195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 413.892767] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:10:44 (1788271844) [ 414.114423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.207119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.314344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 420.302350] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:10:51 (1788271851) [ 420.446843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.498995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.555865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 421.077732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 426.121805] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:10:57 (1788271857) [ 426.287811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.306581] LNet: 23477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.311478] LNet: 23477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 426.329104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.386459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 426.415104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 432.693418] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:11:03 (1788271863) [ 432.841777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.912539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.979767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 433.022182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 439.531357] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:11:10 (1788271870) [ 439.728928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.805874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.857458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 439.885808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 445.274248] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:11:16 (1788271876) [ 445.490331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.534775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.605283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 445.650962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 451.022420] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:11:21 (1788271881) [ 451.129834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.172536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.217383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 451.271325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 451.287416] LNetError: 25834:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 451.324769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 451.347722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 456.469884] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:11:27 (1788271887) [ 456.711751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.779869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.865926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q73tipCuqM [ 457.001585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q73tipCuqM [ 457.081628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q73tipCuqM [ 462.320800] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:11:33 (1788271893) [ 462.529473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.573621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.662524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 462.692291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 462.750609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 468.304932] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:11:39 (1788271899) [ 468.489179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.547446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.625641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 468.647742] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 469.863511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 470.889729] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 472.751243] Key type lgssc registered [ 473.788348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 477.166128] LNetError: 27595:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.48@tcp99 has no available nets [ 479.289404] Key type lgssc unregistered [ 479.595532] LNetError: 28704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 479.605645] LNet: Removed LNI 192.168.203.48@tcp [ 480.142624] Key type .llcrypt unregistered [ 480.145929] Key type ._llcrypt unregistered [ 485.374810] Key type ._llcrypt registered [ 485.378504] Key type .llcrypt registered [ 486.497484] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:11:57 (1788271917) [ 486.925363] Key type .llcrypt unregistered [ 486.927862] Key type ._llcrypt unregistered [ 492.638372] Key type ._llcrypt registered [ 492.641247] Key type .llcrypt registered [ 492.837784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 492.850484] alg: No test for adler32 (adler32-zlib) [ 493.728875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.776343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 493.790811] LNet: 29626:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 493.796110] LNet: 29626:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 493.800779] LNet: 29626:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 493.813756] LNet: Message forwarding will be enabled in 2s [ 495.903119] LNet: Message forwarding enabled [ 496.198168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 496.224599] LNet: Message forwarding will stop in 2s to 6s [ 500.006243] LNet: Message forwarding disabled [ 500.678539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 500.697580] LNet: 29716:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 500.703578] LNet: 29716:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 500.711740] LNet: 29716:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 500.736392] LNet: Message forwarding will be enabled in 2s [ 500.921679] LNet: 29729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.525534] Key type .llcrypt unregistered [ 501.529274] Key type ._llcrypt unregistered [ 512.481626] Key type ._llcrypt registered [ 512.486942] Key type .llcrypt registered [ 514.188726] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:12:24 (1788271944) [ 514.581513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 514.613949] alg: No test for adler32 (adler32-zlib) [ 515.537864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.624633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 521.188705] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:12:31 (1788271951) [ 521.381120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.403618] LNet: 31136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.426553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.496824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 521.528814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 521.560369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 521.607688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 521.637675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 521.701617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 521.749498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 521.819852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 521.874859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 521.912787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 521.940259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 521.965942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 521.989661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 522.016382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 522.050508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 522.079503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 522.103636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 522.127897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 522.155546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 522.171772] LNetError: 31179:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 527.912333] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:12:38 (1788271958) [ 528.231833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.258235] LNet: 31750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.294251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.385634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 534.489667] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:12:45 (1788271965) [ 536.171938] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg348-server [ 538.221996] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:12:48 (1788271968) [ 538.331342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.373629] LNet: 32676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 538.400259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.518299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 538.538517] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 539.540802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[49-53]@tcp [ 539.576740] LNet: 32814:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 539.624736] LNetError: 32684:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.52@tcp is being used as a gateway but routing feature is not turned on [ 539.647169] LNetError: 32684:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.52@tcp has gone from up to down [ 542.688335] LNetError: 32691:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.50@tcp has gone from up to down [ 542.709522] LNetError: 32691:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 545.517197] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:12:56 (1788271976) [ 546.996505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg348-server [ 548.848734] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:12:59 (1788271979) [ 549.034714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.067516] LNet: 33747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.080911] LNetError: 33747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.138470] LNet: Removed LNI 192.168.203.48@tcp [ 549.176877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.229747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 549.249095] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 549.939969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[49-54/2]@tcp [ 549.964444] LNet: 33867:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 549.972736] LNet: 33867:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 550.005932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[49-54/2]@tcp [ 550.039290] LNetError: 33754:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.49@tcp is being used as a gateway but routing feature is not turned on [ 550.066544] LNetError: 33754:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 550.085293] LNetError: 33754:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.49@tcp has gone from up to down [ 550.100804] LNetError: 33754:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 555.997917] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:13:06 (1788271986) [ 556.128701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.155422] LNet: 34459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.166862] LNetError: 34459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 556.186291] LNet: Removed LNI 192.168.203.48@tcp [ 556.211644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.315512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 556.367729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.395739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.452373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 556.519859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.559849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.622242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 556.705383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.736781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.780532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 556.849904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.885846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.934784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 557.002405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.034905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.089366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 557.177863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.228797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.304624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 557.371656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.400172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.470633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 557.529580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.557805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.588838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 557.647652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.676844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.733256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 557.813162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.852115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.903981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 557.948277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.976083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.020926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 563.151539] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:13:13 (1788271993) [ 564.628345] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg348-server [ 565.992144] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:13:17 (1788271997) [ 567.748237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg348-server [ 569.428740] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:13:20 (1788272000) [ 569.576864] LNet: 35859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.587192] LNet: 35859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 570.231305] Key type .llcrypt unregistered [ 570.235288] Key type ._llcrypt unregistered [ 574.910338] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 575.011182] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 575.016798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 575.023468] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 577.638369] Key type ._llcrypt registered [ 577.640571] Key type .llcrypt registered [ 577.838909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 577.870077] alg: No test for adler32 (adler32-zlib) [ 578.797433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.858496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 578.883755] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 579.570825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 579.591025] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 582.000343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 582.019809] LNet: Removed LNI 10.1.2.3@tcp [ 582.175420] LNet: 36578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.185719] LNetError: 36578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.206205] LNet: Removed LNI 192.168.203.48@tcp [ 583.058775] Key type .llcrypt unregistered [ 583.067821] Key type ._llcrypt unregistered [ 593.861990] Key type ._llcrypt registered [ 593.866901] Key type .llcrypt registered [ 595.147953] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:13:45 (1788272025) [ 595.609485] Key type .llcrypt unregistered [ 595.612331] Key type ._llcrypt unregistered [ 601.187966] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 601.341974] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 601.349455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 601.355349] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 603.863296] Key type ._llcrypt registered [ 603.868639] Key type .llcrypt registered [ 604.013531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 604.022673] alg: No test for adler32 (adler32-zlib) [ 604.932662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.973951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 604.991321] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 605.584958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 605.599784] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 607.447097] LNet: 38077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.460398] LNetError: 38077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 607.469671] LNet: Removed LNI 10.1.2.3@tcp [ 607.480029] LNet: Removed LNI 192.168.203.48@tcp [ 608.056968] Key type .llcrypt unregistered [ 608.060203] Key type ._llcrypt unregistered [ 617.536536] Key type ._llcrypt registered [ 617.541080] Key type .llcrypt registered [ 618.718484] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:14:09 (1788272049) [ 619.313367] Key type .llcrypt unregistered [ 619.321663] Key type ._llcrypt unregistered [ 623.922659] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 623.976757] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 623.979057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 623.981798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 626.432166] Key type ._llcrypt registered [ 626.434298] Key type .llcrypt registered [ 626.543222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 626.558473] alg: No test for adler32 (adler32-zlib) [ 627.435498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.497900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 627.522325] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 628.963839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 628.984300] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 630.397265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 630.415521] LNet: Removed LNI 10.1.2.3@tcp [ 630.426650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 630.452213] LNet: Removed LNI 10.1.2.31@tcp [ 630.466760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 630.484730] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 631.040447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 631.059788] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 631.732333] LNet: 40004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.746565] LNetError: 40004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 631.753439] LNet: Removed LNI 10.1.2.31@tcp [ 631.766303] LNet: Removed LNI 10.1.2.3@tcp [ 632.276478] Key type .llcrypt unregistered [ 632.279332] Key type ._llcrypt unregistered [ 640.605697] Key type ._llcrypt registered [ 640.609132] Key type .llcrypt registered [ 641.452537] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:14:32 (1788272072) [ 642.005425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 642.017387] alg: No test for adler32 (adler32-zlib) [ 642.877533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.932567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 642.947311] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 643.433762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.452404] LNet: 40960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.458452] LNetError: 40960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.480318] LNet: Removed LNI 192.168.203.48@tcp [ 643.496225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.544983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.48@tcp [ 643.564158] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 644.083745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.48@tcp [ 644.106507] LNet: Removed LNI 192.168.203.48@tcp [ 644.158969] LNet: 41095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.706526] Key type .llcrypt unregistered [ 644.708345] Key type ._llcrypt unregistered [ 653.812044] Key type ._llcrypt registered [ 653.814621] Key type .llcrypt registered [ 654.743388] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:14:45 (1788272085) [ 656.096314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg348-server [ 657.396181] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:14:48 (1788272088) [ 657.766284] Key type .llcrypt unregistered [ 657.769244] Key type ._llcrypt unregistered [ 662.702239] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 662.783495] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 662.794902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 662.804673] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 665.409417] Key type ._llcrypt registered [ 665.411538] Key type .llcrypt registered [ 665.605097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 665.625454] alg: No test for adler32 (adler32-zlib) [ 666.560956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.607651] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 667.207266] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 668.572686] LNet: 42935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.581662] LNetError: 42935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.588292] LNet: Removed LNI 192.168.203.48@tcp1 [ 668.600730] LNet: Removed LNI 10.1.2.3@tcp2 [ 669.093976] Key type .llcrypt unregistered [ 669.096925] Key type ._llcrypt unregistered [ 678.051698] Key type ._llcrypt registered [ 678.056482] Key type .llcrypt registered [ 679.195954] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:15:10 (1788272110) [ 679.579521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 679.588803] alg: No test for adler32 (adler32-zlib) [ 680.495170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.555901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 680.582278] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 681.270077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.287470] LNet: 43909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.294536] LNetError: 43909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.309402] LNet: Removed LNI 192.168.203.48@tcp [ 681.338582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.412875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 681.431838] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 687.262973] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:15:18 (1788272118) [ 687.459559] LNet: 44502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.467449] LNetError: 44502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.491213] LNet: Removed LNI 192.168.203.48@tcp [ 688.022414] Key type .llcrypt unregistered [ 688.024474] Key type ._llcrypt unregistered [ 692.604094] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 692.722221] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 692.724522] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 692.728390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 695.170790] Key type ._llcrypt registered [ 695.173558] Key type .llcrypt registered [ 695.366578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.377443] alg: No test for adler32 (adler32-zlib) [ 696.238798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.283806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.303900] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 696.837361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 696.855462] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 698.695994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.710710] LNet: 45247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.715667] LNetError: 45247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 698.720435] LNet: Removed LNI 10.1.2.3@tcp [ 698.734247] LNet: Removed LNI 192.168.203.48@tcp [ 698.746500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.797122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 698.811642] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 698.815775] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 699.064634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.093900] LNet: Removed LNI 10.1.2.3@tcp [ 699.124082] LNet: Removed LNI 192.168.203.48@tcp [ 699.151183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.184857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 699.199812] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 699.202556] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 699.482702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 699.505660] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 700.330534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.369046] LNet: 45436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.378792] LNet: 45436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 700.393449] LNetError: 45436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 700.397884] LNetError: 45436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 700.403917] LNet: Removed LNI 10.1.2.3@tcp [ 700.406288] LNet: Removed LNI 192.168.203.48@tcp [ 700.426322] LNet: Removed LNI 192.168.203.48@tcp2 [ 700.466875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.546783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 700.576080] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 700.579585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.586426] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 705.807366] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:15:36 (1788272136) [ 705.968261] LNet: 46034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.976840] LNetError: 46034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 705.983825] LNetError: 46034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 705.997489] LNet: Removed LNI 10.1.2.3@tcp [ 705.999242] LNet: Removed LNI 192.168.203.48@tcp [ 706.015112] LNet: Removed LNI 192.168.203.48@tcp2 [ 706.602167] Key type .llcrypt unregistered [ 706.608606] Key type ._llcrypt unregistered [ 711.854804] Key type ._llcrypt registered [ 711.860738] Key type .llcrypt registered [ 712.048948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 712.060799] alg: No test for adler32 (adler32-zlib) [ 712.918270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.983020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.007620] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 713.652669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.674984] LNet: 46433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.681843] LNetError: 46433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.695538] LNet: Removed LNI 192.168.203.48@tcp [ 713.713566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.762180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 713.781243] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 714.147177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.179894] LNet: 46463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.192355] LNetError: 46463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 714.211484] LNet: Removed LNI 192.168.203.48@tcp [ 714.246690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.309653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 714.333994] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 714.673248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.692856] LNet: Removed LNI 192.168.203.48@tcp [ 714.711324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.778637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 714.816686] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 715.206732] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 715.321712] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 715.328467] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 715.338625] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 717.691813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.713751] LNet: 46571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.718953] LNet: 46571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 717.729989] LNetError: 46571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.735613] LNetError: 46571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 717.755175] LNet: Removed LNI 192.168.203.48@tcp [ 717.771524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.829508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 717.851419] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 722.750304] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:15:53 (1788272153) [ 722.891655] LNet: 47168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.897119] LNetError: 47168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.911950] LNet: Removed LNI 192.168.203.48@tcp [ 723.328644] Key type .llcrypt unregistered [ 723.330863] Key type ._llcrypt unregistered [ 728.430265] Key type ._llcrypt registered [ 728.433728] Key type .llcrypt registered [ 728.646227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 728.658621] alg: No test for adler32 (adler32-zlib) [ 729.512439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.549893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 729.575089] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 730.217744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.235887] LNet: 47570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.244328] LNetError: 47570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.254861] LNet: Removed LNI 192.168.203.48@tcp [ 730.282872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.327771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 730.347908] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 735.040914] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:16:06 (1788272166) [ 735.124312] LNet: 48155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.130226] LNetError: 48155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.156825] LNet: Removed LNI 192.168.203.48@tcp [ 735.708459] Key type .llcrypt unregistered [ 735.710221] Key type ._llcrypt unregistered [ 739.778706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 739.861984] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 739.867094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 739.873664] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.315971] Key type ._llcrypt registered [ 742.318408] Key type .llcrypt registered [ 742.505880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.514674] alg: No test for adler32 (adler32-zlib) [ 743.356538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.385457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.399149] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 743.950906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.966970] LNet: 48616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.972892] LNetError: 48616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.988598] LNet: Removed LNI 192.168.203.48@tcp [ 744.002761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.057220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 744.076845] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 748.713323] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:16:19 (1788272179) [ 748.890113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.912312] LNet: 49213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.923328] LNetError: 49213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.937780] LNet: Removed LNI 192.168.203.48@tcp [ 748.956285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.057311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 754.085718] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:16:25 (1788272185) [ 754.173948] LNet: 49788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.617861] Key type .llcrypt unregistered [ 754.622489] Key type ._llcrypt unregistered [ 758.960665] Key type ._llcrypt registered [ 758.964322] Key type .llcrypt registered [ 759.106761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.119674] alg: No test for adler32 (adler32-zlib) [ 759.976419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.021027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 760.038819] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 760.626766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.647611] LNet: 50197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.657514] LNetError: 50197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.672256] LNet: Removed LNI 192.168.203.48@tcp [ 760.692217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.737645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 760.751262] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 765.006221] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:16:36 (1788272196) [ 765.130918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.163581] LNet: 50789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.170762] LNetError: 50789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.182894] LNet: Removed LNI 192.168.203.48@tcp [ 765.202936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.257756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 769.191208] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:16:40 (1788272200) [ 770.180340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 771.357737] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:16:42 (1788272202) [ 772.275322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 773.479121] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:16:44 (1788272204) [ 774.582377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 775.829879] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:16:46 (1788272206) [ 776.876712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 777.886419] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:16:49 (1788272209) [ 778.890996] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 779.862558] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:16:51 (1788272211) [ 780.017239] LNet: 52841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.448845] Key type .llcrypt unregistered [ 780.450849] Key type ._llcrypt unregistered [ 783.860803] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 784.010555] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 784.014145] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 784.019055] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 786.390803] Key type ._llcrypt registered [ 786.392654] Key type .llcrypt registered [ 786.551869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 786.574697] alg: No test for adler32 (adler32-zlib) [ 787.452527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.527846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 787.545574] LNet: Added LNI 192.168.203.48@tcp [8/512/0/180] [ 787.553689] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 787.556089] LNet: 53166:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 787.630457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.653646] LNet: 53185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.660910] LNetError: 53185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 787.668904] LNet: Removed LNI 10.1.2.3@tcp [ 787.670744] LNetError: 53185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.686394] LNet: Removed LNI 192.168.203.48@tcp [ 787.710464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.758996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 787.776720] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 787.829929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.849704] LNet: 53209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.855787] LNetError: 53209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.868730] LNet: Removed LNI 192.168.203.48@tcp [ 787.884452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.930614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 787.961488] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 787.969329] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 787.972516] LNet: 53217:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 788.034900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.048093] LNet: 53235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.052643] LNetError: 53235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 788.061061] LNet: Removed LNI 10.1.2.3@tcp [ 788.063926] LNetError: 53235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.077075] LNet: Removed LNI 192.168.203.48@tcp [ 788.092449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.146618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 788.171485] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 788.178516] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 788.182351] LNet: 53243:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 788.316767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.337918] LNet: 53269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.343148] LNetError: 53269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 788.349603] LNet: Removed LNI 10.1.2.3@tcp [ 788.352499] LNetError: 53269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.365217] LNet: Removed LNI 192.168.203.48@tcp [ 788.378875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.429499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 788.448829] LNet: Added LNI 192.168.203.48@tcp [8/256/0/182] [ 788.454625] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 788.544924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.569301] LNet: 53297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.575831] LNetError: 53297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 788.585469] LNet: Removed LNI 10.1.2.3@tcp [ 788.590975] LNetError: 53297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.611573] LNet: Removed LNI 192.168.203.48@tcp [ 788.640344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.699558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 788.718790] LNet: Added LNI 192.168.203.48@tcp [8/256/0/182] [ 788.726698] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 788.729053] LNet: 53305:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 788.781755] LNet: Removed LNI 10.1.2.3@tcp [ 788.797282] LNet: Removed LNI 192.168.203.48@tcp [ 789.212389] Key type .llcrypt unregistered [ 789.215355] Key type ._llcrypt unregistered [ 797.860255] Key type ._llcrypt registered [ 797.864872] Key type .llcrypt registered [ 798.881319] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:17:09 (1788272229) [ 799.985376] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 801.313623] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:17:12 (1788272232) [ 801.754785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 801.764788] alg: No test for adler32 (adler32-zlib) [ 802.634501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.672560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 802.701845] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 803.234481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.48 [ 803.247225] LNet: 54572:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 803.254277] LNet: Removed LNI 192.168.203.48@tcp [ 807.341628] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:17:18 (1788272238) [ 807.591348] LNet: 55142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.604068] LNetError: 55142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.621107] LNet: Removed LNI 192.168.203.48@tcp [ 808.114724] Key type .llcrypt unregistered [ 808.117416] Key type ._llcrypt unregistered [ 812.488636] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 812.667978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 815.330750] Key type ._llcrypt registered [ 815.334183] Key type .llcrypt registered [ 815.451669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 815.474698] alg: No test for adler32 (adler32-zlib) [ 816.690658] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 821.388233] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:17:32 (1788272252) [ 821.491870] LNet: 56055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.499939] LNetError: 56055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 821.515138] LNet: Removed LNI 10.1.2.3@tcp [ 821.929676] Key type .llcrypt unregistered [ 821.931899] Key type ._llcrypt unregistered [ 825.723088] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 825.897877] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 825.901531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 825.905934] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 828.412329] Key type ._llcrypt registered [ 828.415490] Key type .llcrypt registered [ 828.523149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 828.538466] alg: No test for adler32 (adler32-zlib) [ 829.439297] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 833.978271] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:17:45 (1788272265) [ 835.131465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 836.456864] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:17:47 (1788272267) [ 836.578823] LNet: 57246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.601388] LNetError: 57246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 836.623154] LNet: Removed LNI 10.1.2.3@tcp [ 837.056583] Key type .llcrypt unregistered [ 837.059684] Key type ._llcrypt unregistered [ 841.170588] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 841.313752] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 841.323101] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 841.328904] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 843.914336] Key type ._llcrypt registered [ 843.916684] Key type .llcrypt registered [ 844.062755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 844.078579] alg: No test for adler32 (adler32-zlib) [ 844.944783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.998696] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 850.492689] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:18:01 (1788272281) [ 850.908467] LNet: 58480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.915223] LNetError: 58480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 850.926184] LNet: Removed LNI 10.1.2.3@tcp [ 851.318632] Key type .llcrypt unregistered [ 851.323363] Key type ._llcrypt unregistered [ 854.850694] Key type ._llcrypt registered [ 854.853293] Key type .llcrypt registered [ 854.979922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 854.992651] alg: No test for adler32 (adler32-zlib) [ 855.896982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 855.915227] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 862.412923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 863.699703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_interrupt -r 1 [ 863.761287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_interrupt -r 1 [ 863.822698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 863.858711] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 864.168115] LNet: 1 local NIs in recovery (showing 1): 192.168.203.48@tcp [ 864.277538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 864.308968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 864.361678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 864.434661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 864.481509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 864.604883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 864.708194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 864.837577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 864.895147] LNet: 58921:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 864.968522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 865.017048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 865.078780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 865.143814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 865.208266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 865.269557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 865.315513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 865.889248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_dropped -r 1 [ 865.921873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_dropped -r 1 [ 865.948784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 865.964549] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 865.969236] LNet: Skipped 1 previous similar message [ 866.204476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 866.278602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 866.324636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 866.395580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 866.450240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 866.536829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 866.580191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 866.609931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 866.630555] LNet: 59031:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 866.650700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 866.688664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 866.728280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 866.765704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 866.798984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 866.828621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 866.862624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 868.388729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_aborted -r 1 [ 868.520956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_aborted -r 1 [ 868.619344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 868.688207] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 868.706978] LNet: Skipped 1 previous similar message [ 869.099763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 869.125089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 869.158878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 869.180456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 869.209514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 869.239798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 869.281212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 869.309120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 869.331233] LNet: 59140:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 869.358746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 869.398173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 869.430944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 869.465255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 869.498670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 869.523947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 869.546725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 870.520647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_no_route -r 1 [ 870.602560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_no_route -r 1 [ 870.628856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 870.881574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 870.902787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 870.935600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 870.983168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 871.014457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 871.036866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 871.060736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 871.084158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 871.110898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 871.135841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 871.161446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 871.188523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 871.213791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 871.247783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 871.270504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 871.786973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_timeout -r 1 [ 871.855628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_timeout -r 1 [ 871.879805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 871.896830] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 871.902805] LNet: Skipped 1 previous similar message [ 872.137507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 872.184379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 872.258292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 872.309959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 872.395426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 872.456706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 872.491191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 872.521239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 872.538187] LNet: 59359:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 872.548102] LNet: 59359:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 872.564943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 872.595603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 872.629932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 872.678501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 872.723934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 872.766845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 872.811449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 873.383409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_error -r 1 [ 873.457054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_error -r 1 [ 873.615169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 873.978513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 874.011094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 874.039784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 874.071698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 874.097585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 874.122579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 874.143350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 874.168735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 874.190751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 874.229719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 874.263209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 874.298828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 874.344910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 874.388432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 874.422516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 874.907907] LNet: 59513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.919856] LNetError: 59513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.941824] LNet: Removed LNI 192.168.203.48@tcp [ 876.434491] Key type .llcrypt unregistered [ 876.440959] Key type ._llcrypt unregistered [ 905.501451] Key type ._llcrypt registered [ 905.506682] Key type .llcrypt registered [ 907.150069] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:18:57 (1788272337) [ 908.595938] Key type .llcrypt unregistered [ 908.598505] Key type ._llcrypt unregistered [ 915.308706] Key type ._llcrypt registered [ 915.311480] Key type .llcrypt registered [ 915.518798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 915.536563] alg: No test for adler32 (adler32-zlib) [ 916.507687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 916.533751] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 923.533577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 936.436943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.527105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 936.580543] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 939.117721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_interrupt -r 1 [ 939.173576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_interrupt -r 1 [ 939.220777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_interrupt -r 1 [ 939.313133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_interrupt -r 1 [ 939.394687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_interrupt -r 1 [ 939.447964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_interrupt -r 1 [ 939.516229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_interrupt -r 1 [ 939.575761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_interrupt -r 1 [ 939.658534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 939.678220] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 939.691094] LNet: 1 local NIs in recovery (showing 1): 192.168.203.48@tcp [ 940.164095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 940.229275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 940.275289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 940.361429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 940.415350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 940.503696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 940.594779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 940.672261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 940.708835] LNet: There was an unexpected network error while writing to 192.168.203.48: rc = -22 [ 940.714460] LNet: 60991:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 940.722409] LNet: Skipped 3 previous similar messages [ 940.749384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 940.821898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 940.885451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 940.922114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 940.957578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 941.058759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 941.095301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 941.961122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_dropped -r 1 [ 942.007938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_dropped -r 1 [ 942.079727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_dropped -r 1 [ 942.171766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_dropped -r 1 [ 942.221682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_dropped -r 1 [ 942.305172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_dropped -r 1 [ 942.345660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_dropped -r 1 [ 942.388414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_dropped -r 1 [ 942.408904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 942.440693] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 942.798261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 942.872558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 942.930299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 942.994472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 943.071425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 943.184124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 943.250962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 943.424424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 943.509340] LNet: 61113:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 943.559755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 943.666647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 943.747546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 943.850381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 943.917483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 943.961971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 943.986425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 944.857696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_aborted -r 1 [ 944.963574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_aborted -r 1 [ 945.091994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_aborted -r 1 [ 945.157710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_aborted -r 1 [ 945.266630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_aborted -r 1 [ 945.372889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_aborted -r 1 [ 945.489197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_aborted -r 1 [ 945.568110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_aborted -r 1 [ 945.682422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 945.736363] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 945.749882] LNet: Skipped 2 previous similar messages [ 946.177258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 946.274522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 946.345379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 946.424486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 946.520349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 946.597292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 946.668621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 946.753892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 946.818241] LNet: 61234:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 946.865812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 946.948957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 947.029413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 947.168365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 947.242078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 947.318734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 947.422684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 948.333450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_no_route -r 1 [ 948.382519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_no_route -r 1 [ 948.422528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_no_route -r 1 [ 948.505473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_no_route -r 1 [ 948.567862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_no_route -r 1 [ 948.650893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_no_route -r 1 [ 948.773663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_no_route -r 1 [ 948.906615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_no_route -r 1 [ 948.974777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 949.408865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 949.451625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 949.504058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 949.567903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 949.619669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 949.658507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 949.703227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 949.783940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 949.819469] LNet: 61356:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 949.864491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 949.963751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 950.073668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 950.183597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 950.282915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 950.347560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 950.425375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 951.411493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_timeout -r 1 [ 951.493295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_timeout -r 1 [ 951.560602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_timeout -r 1 [ 951.637498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_timeout -r 1 [ 951.718864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_timeout -r 1 [ 951.799511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_timeout -r 1 [ 951.870083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_timeout -r 1 [ 951.925819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_timeout -r 1 [ 951.994483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 952.019591] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 952.029978] LNet: Skipped 5 previous similar messages [ 952.331701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 952.384229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 952.409627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 952.440637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 952.491097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 952.541375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 952.626774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 952.680474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 952.758271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 952.795802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 952.863684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 952.944381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 952.998358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 953.034545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 953.073894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 953.658967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_error -r 1 [ 953.706191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_error -r 1 [ 953.774176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_error -r 1 [ 953.816768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_error -r 1 [ 953.870510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_error -r 1 [ 953.905618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_error -r 1 [ 953.968546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_error -r 1 [ 954.057167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_error -r 1 [ 954.122232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 954.439197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 954.475846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 954.505674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 954.561347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 954.608363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 954.696303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 954.735406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 954.784986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 954.816641] LNet: 61599:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 954.828626] LNet: 61599:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 954.846858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 954.885512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 954.917376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 954.942213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 954.969035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 955.021877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 955.065644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 961.678875] LNet: 61679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.687607] LNetError: 61679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.719248] LNet: Removed LNI 192.168.203.48@tcp [ 962.725688] LNetError: 61679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.753504] LNet: Removed LNI 192.168.203.48@tcp1 [ 963.166984] Key type .llcrypt unregistered [ 963.173454] Key type ._llcrypt unregistered [ 980.230769] Key type ._llcrypt registered [ 980.233705] Key type .llcrypt registered [ 982.583410] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:20:12 (1788272412) [ 983.998572] Key type .llcrypt unregistered [ 984.004372] Key type ._llcrypt unregistered [ 992.073959] Key type ._llcrypt registered [ 992.078383] Key type .llcrypt registered [ 992.321531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 992.349207] alg: No test for adler32 (adler32-zlib) [ 993.291330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 993.314663] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1001.165042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 1002.019968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e remote_dropped -r 1 [ 1002.067949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e remote_dropped -r 1 [ 1002.147157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 1002.172944] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1002.593476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1002.632486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1002.666258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1002.724745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1002.794354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1002.861783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1002.910792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1002.965894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1003.015657] LNet: 62934:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1003.046283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1003.108975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1003.198738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1003.258602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1003.331891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1003.402178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1003.475866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1004.172372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e remote_error -r 1 [ 1004.208574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e remote_error -r 1 [ 1004.270875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 1004.307425] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1004.675736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1004.754279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1004.802846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1004.882086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1004.924490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1004.992253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1005.071193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1005.127781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1005.158224] LNet: 63043:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1005.202408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1005.243581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1005.308672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1005.369622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1005.420791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1005.483296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1005.528258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1006.089390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e remote_timeout -r 1 [ 1006.139857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e remote_timeout -r 1 [ 1006.215848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 1006.239756] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1006.553434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1006.590028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1006.631098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1006.684268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1006.717713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1006.778299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1006.823811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1006.906697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1006.946476] LNet: 63152:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1006.980099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1007.027395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1007.075513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1007.129734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1007.164161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1007.211444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1007.263740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1007.552250] LNet: 63197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.558628] LNetError: 63197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.573759] LNet: Removed LNI 192.168.203.48@tcp [ 1008.074482] Key type .llcrypt unregistered [ 1008.076052] Key type ._llcrypt unregistered [ 1023.566788] Key type ._llcrypt registered [ 1023.569434] Key type .llcrypt registered [ 1024.725888] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:20:55 (1788272455) [ 1025.719382] Key type .llcrypt unregistered [ 1025.721960] Key type ._llcrypt unregistered [ 1031.983618] Key type ._llcrypt registered [ 1031.989875] Key type .llcrypt registered [ 1032.168127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1032.186491] alg: No test for adler32 (adler32-zlib) [ 1033.141391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1033.169166] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1040.482859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 1049.811270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1049.872760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1049.912769] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1051.330094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e remote_dropped -r 1 [ 1051.390224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e remote_dropped -r 1 [ 1051.456914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e remote_dropped -r 1 [ 1051.528392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e remote_dropped -r 1 [ 1051.586682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e remote_dropped -r 1 [ 1051.661739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e remote_dropped -r 1 [ 1051.706345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e remote_dropped -r 1 [ 1051.752438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e remote_dropped -r 1 [ 1051.791511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 1051.819887] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1051.836905] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.148@tcp [ 1052.054924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.083412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.114328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.153387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.210767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.256018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.291305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.343712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.373421] LNet: 64668:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1052.404455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.431864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.478952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.509827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.542801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.573723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.600029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1052.634949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.239601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e remote_error -r 1 [ 1053.276919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e remote_error -r 1 [ 1053.316534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e remote_error -r 1 [ 1053.357116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e remote_error -r 1 [ 1053.424631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e remote_error -r 1 [ 1053.490237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e remote_error -r 1 [ 1053.527073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e remote_error -r 1 [ 1053.566975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e remote_error -r 1 [ 1053.628076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 1053.663740] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1053.674707] LNet: Skipped 2 previous similar messages [ 1053.913093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.940678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1053.962907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.984579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.017366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.048890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.076188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.105498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.125268] LNet: 64791:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1054.167696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.198482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.230455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.283186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.329160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.374543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.410214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1054.441320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.984523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e remote_timeout -r 1 [ 1055.015892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e remote_timeout -r 1 [ 1055.060334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e remote_timeout -r 1 [ 1055.103889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e remote_timeout -r 1 [ 1055.168583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e remote_timeout -r 1 [ 1055.215845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e remote_timeout -r 1 [ 1055.261451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e remote_timeout -r 1 [ 1055.323613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e remote_timeout -r 1 [ 1055.383590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.148@tcp [ 1055.421595] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1055.679952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.723563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1055.766984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.803751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.842610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.891953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.940243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.993992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1056.026971] LNet: 64914:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1056.051774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1056.084585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.121276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.166386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.207550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.239145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.296274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1056.344497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1061.139988] LNet: 64996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.149601] LNetError: 64996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.165798] LNet: Removed LNI 192.168.203.48@tcp [ 1061.180727] LNet: Removed LNI 192.168.203.48@tcp1 [ 1061.681718] Key type .llcrypt unregistered [ 1061.685488] Key type ._llcrypt unregistered [ 1079.273934] Key type ._llcrypt registered [ 1079.277785] Key type .llcrypt registered [ 1080.308710] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:21:51 (1788272511) [ 1081.826982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1083.265483] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:21:54 (1788272514) [ 1084.509531] Key type .llcrypt unregistered [ 1084.513136] Key type ._llcrypt unregistered [ 1091.814177] Key type ._llcrypt registered [ 1091.819375] Key type .llcrypt registered [ 1092.167568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.193394] alg: No test for adler32 (adler32-zlib) [ 1093.132716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1093.171100] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1100.787435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 1101.473084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e network_timeout -r 1 [ 1101.514293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e network_timeout -r 1 [ 1101.557495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 1101.584627] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1101.858816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.894330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.930431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.977576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.016153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.073126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.116802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.147795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.168283] LNet: 66543:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.192948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.225608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.250887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.289326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.329438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.379781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.735925] LNet: 66586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.744092] LNetError: 66586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1102.757785] LNet: Removed LNI 192.168.203.48@tcp [ 1103.158648] Key type .llcrypt unregistered [ 1103.161344] Key type ._llcrypt unregistered [ 1120.735603] Key type ._llcrypt registered [ 1120.740851] Key type .llcrypt registered [ 1121.008058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1121.024995] alg: No test for adler32 (adler32-zlib) [ 1121.921915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1121.951645] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1129.128604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 1137.969625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1138.007179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1138.029694] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1139.378296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e network_timeout -r 1 [ 1139.424497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e network_timeout -r 1 [ 1139.471494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e network_timeout -r 1 [ 1139.519707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e network_timeout -r 1 [ 1139.574767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e network_timeout -r 1 [ 1139.601819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e network_timeout -r 1 [ 1139.640611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e network_timeout -r 1 [ 1139.683313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e network_timeout -r 1 [ 1139.710289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 1139.732464] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1139.890949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.910650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1139.938555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.979687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.029772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.072571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.126720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.158793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.194025] LNet: 67468:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.216744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.242348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.284933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.308367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.337713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.363301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.811803] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1145.259596] LNet: 67546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.271840] LNetError: 67546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.279607] LNet: Removed LNI 192.168.203.48@tcp [ 1145.289467] LNet: Removed LNI 192.168.203.48@tcp1 [ 1145.847200] Key type .llcrypt unregistered [ 1145.849776] Key type ._llcrypt unregistered [ 1163.374124] Key type ._llcrypt registered [ 1163.376202] Key type .llcrypt registered [ 1164.918159] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:23:15 (1788272595) [ 1165.352940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1165.367798] alg: No test for adler32 (adler32-zlib) [ 1166.330327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1166.375336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1166.400418] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1167.046233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1167.067983] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1167.770652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1167.851214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1167.914182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1167.945368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1167.972666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1168.003282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1168.031777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.48@tcp [ 1168.051072] LNetError: 68646:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.48@tcp to recovery [ 1168.416261] LNet: 1 local NIs in recovery (showing 1): 192.168.203.48@tcp [ 1168.420649] LNet: There was an unexpected network error while writing to 192.168.203.48: rc = -22 [ 1170.464255] LNet: There was an unexpected network error while writing to 192.168.203.48: rc = -22 [ 1171.637766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1174.559405] LNet: There was an unexpected network error while writing to 192.168.203.48: rc = -22 [ 1175.280774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1175.336647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.421981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1175.440484] LNet: 68753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.447626] LNetError: 68753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1176.479885] LNet: Removed LNI 192.168.203.48@tcp [ 1176.485113] LNetError: 68753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1176.516489] LNet: Removed LNI 192.168.203.48@tcp1 [ 1176.553815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1176.654439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1176.676643] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1177.322483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1177.350838] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1178.418380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1178.462541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.542274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1178.585324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1178.629238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1178.673503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1178.718515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1178.755525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.48@tcp [ 1178.777966] LNetError: 69026:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.48@tcp to recovery [ 1179.679496] LNet: There was an unexpected network error while writing to 192.168.203.48: rc = -22 [ 1182.276802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1185.823346] LNet: There was an unexpected network error while writing to 192.168.203.48: rc = -22 [ 1185.842897] LNet: Skipped 1 previous similar message [ 1190.228800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1190.273696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1190.329583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.281198] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:23:47 (1788272627) [ 1197.363646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1197.381735] LNet: 69735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.391829] LNetError: 69735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.399701] LNet: Removed LNI 192.168.203.48@tcp [ 1197.414580] LNet: Removed LNI 192.168.203.48@tcp1 [ 1197.443624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1197.555380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1197.594932] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1198.352975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1198.376721] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1199.070950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1199.134074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1199.161635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1199.190233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1199.239317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1199.296070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1199.339210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.48@tcp --health 0 [ 1199.441294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1199.529240] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.48@tcp [ 1200.545087] LNet: There was an unexpected network error while writing to 192.168.203.48: rc = -22 [ 1200.556367] LNet: Skipped 1 previous similar message [ 1215.035824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1215.205890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1215.258139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.288241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.48@tcp --health 500 [ 1215.398904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1222.116705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1222.162555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1222.194968] LNet: 70235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.205951] LNetError: 70235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.213807] LNetError: 70235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1222.230489] LNet: Removed LNI 192.168.203.48@tcp [ 1222.260610] LNet: Removed LNI 192.168.203.48@tcp1 [ 1222.304486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1222.364910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1222.396459] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1223.143310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1223.166150] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1223.964824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1224.033209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.113487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1224.152555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1224.185267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1224.215650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1224.248982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1224.291633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.48@tcp --health 0 [ 1225.375333] LNet: There was an unexpected network error while writing to 192.168.203.48: rc = -22 [ 1225.390368] LNet: Skipped 3 previous similar messages [ 1225.640619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1236.173301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1236.207610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.248593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.615127] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:24:34 (1788272674) [ 1244.069739] LNet: 71281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1244.076491] LNetError: 71281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1244.080158] LNetError: 71281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1244.086515] LNet: Removed LNI 192.168.203.48@tcp [ 1244.103572] LNet: Removed LNI 192.168.203.48@tcp1 [ 1244.697910] Key type .llcrypt unregistered [ 1244.702952] Key type ._llcrypt unregistered [ 1250.561789] Key type ._llcrypt registered [ 1250.563424] Key type .llcrypt registered [ 1250.751837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1250.763725] alg: No test for adler32 (adler32-zlib) [ 1251.701452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1251.727971] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1260.289440] Lustre: DEBUG MARKER: Initial discovery [ 1260.342583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.148@tcp [ 1263.208359] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1263.311920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1263.401933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.148@tcp [ 1263.434680] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1263.466447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1263.538807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.148@tcp -r 1 -m GET -l 3 [ 1263.588906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.148@tcp -r 1 -m GET -e local_error [ 1263.644870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.148@tcp -r 1 -m PUT -l 6 [ 1264.033357] LNet: 1 local NIs in recovery (showing 1): 192.168.203.48@tcp [ 1265.086660] Lustre: DEBUG MARKER: Force oleg348-client.virtnet to discover 192.168.203.148@tcp (in background) [ 1265.379487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.148@tcp [ 1266.701566] Lustre: DEBUG MARKER: Force oleg348-server to discover 192.168.203.48@tcp [ 1268.447201] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1269.158940] Lustre: DEBUG MARKER: Wait for 72049 [ 1270.862965] Lustre: DEBUG MARKER: Finished wait on 72049 [ 1270.954448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1270.993206] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1271.021111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1271.415585] LNet: 72501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1271.421399] LNetError: 72501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1271.446433] LNet: Removed LNI 192.168.203.48@tcp [ 1271.964564] Key type .llcrypt unregistered [ 1271.967335] Key type ._llcrypt unregistered [ 1288.008473] Key type ._llcrypt registered [ 1288.010812] Key type .llcrypt registered [ 1289.266527] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:25:19 (1788272719) [ 1289.907461] Key type .llcrypt unregistered [ 1289.909754] Key type ._llcrypt unregistered [ 1296.691286] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1296.820879] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1296.827994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1296.844080] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1299.781454] Key type ._llcrypt registered [ 1299.785315] Key type .llcrypt registered [ 1300.060458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1300.076245] alg: No test for adler32 (adler32-zlib) [ 1301.115467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1301.205248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1301.234641] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1302.271938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1302.321681] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1311.366844] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:25:41 (1788272741) [ 1311.511262] LNet: 74575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.521604] LNetError: 74575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1311.535974] LNet: Removed LNI 10.1.2.3@tcp [ 1311.549080] LNet: Removed LNI 192.168.203.48@tcp [ 1312.473663] Key type .llcrypt unregistered [ 1312.478808] Key type ._llcrypt unregistered [ 1319.880193] Key type ._llcrypt registered [ 1319.881930] Key type .llcrypt registered [ 1320.161147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1320.192289] alg: No test for adler32 (adler32-zlib) [ 1321.171557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1321.231944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1321.263420] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1321.970632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1322.013796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1322.942870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.48@tcp [ 1324.000318] LNetError: Unexpected error -22 connecting to 192.168.203.48@tcp at host 192.168.203.48:988 [ 1329.473758] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:25:59 (1788272759) [ 1329.542456] LNet: 75718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.551249] LNetError: 75718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1329.568449] LNet: Removed LNI 10.1.2.3@tcp [ 1329.593585] LNet: Removed LNI 192.168.203.48@tcp [ 1330.195668] Key type .llcrypt unregistered [ 1330.204527] Key type ._llcrypt unregistered [ 1336.303849] Key type ._llcrypt registered [ 1336.307292] Key type .llcrypt registered [ 1336.533111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1336.559394] alg: No test for adler32 (adler32-zlib) [ 1337.518496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.611541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1337.656591] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1338.569184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1338.586483] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 1339.582268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.48@tcp1 --nid 192.168.203.48@tcp2 [ 1343.388799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.48@tcp1 192.168.203.48@tcp2 [ 1343.418765] LNetError: 76566:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.48@tcp1 to 192.168.203.48@tcp2 - Route table may be misconfigured [ 1343.448755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.48@tcp2 192.168.203.48@tcp1 [ 1349.581473] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:26:20 (1788272780) [ 1349.692951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1349.737332] LNet: 77134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.750322] LNetError: 77134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.760473] LNet: Removed LNI 192.168.203.48@tcp1 [ 1350.823991] LNet: Removed LNI 192.168.203.48@tcp2 [ 1350.869640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1350.945470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1350.970097] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1351.574213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1351.592455] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1352.438695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1352.474690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.48@tcp 192.168.203.48@tcp [ 1352.545377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.48@tcp1 192.168.203.48@tcp1 [ 1352.581624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.48@tcp -d 192.168.203.48@tcp -e network_timeout [ 1352.607871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -e network_timeout [ 1352.650985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -e network_timeout [ 1352.681571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -e network_timeout [ 1352.708467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1352.729830] LNet: There was an unexpected network error while writing to 192.168.203.48: rc = -22 [ 1352.752342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1352.882301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1352.991649] LNet: 1 local NIs in recovery (showing 1): 192.168.203.48@tcp [ 1353.004710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1359.420655] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:26:29 (1788272789) [ 1359.540936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1359.607723] LNet: 77997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.620441] LNetError: 77997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.635988] LNetError: 77997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1359.648447] LNet: Removed LNI 192.168.203.48@tcp [ 1360.689689] LNet: Removed LNI 192.168.203.48@tcp1 [ 1360.784746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.881956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1361.253705] LNet: 78044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.686680] Key type .llcrypt unregistered [ 1361.693403] Key type ._llcrypt unregistered [ 1372.654235] Key type ._llcrypt registered [ 1372.659126] Key type .llcrypt registered [ 1373.990838] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:26:44 (1788272804) [ 1374.326256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1374.338956] alg: No test for adler32 (adler32-zlib) [ 1375.236356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.833312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1378.969159] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:26:49 (1788272809) [ 1379.121799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.206743] LNet: 79138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.237786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.350828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1379.384633] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1380.436681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1380.477975] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1381.592893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1381.649894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp1 [ 1381.686988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp1 [ 1387.288908] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:26:58 (1788272818) [ 1388.858920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg348-server [ 1390.667761] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:27:01 (1788272821) [ 1392.293486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg348-server [ 1393.680181] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:27:04 (1788272824) [ 1395.296512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg348-server [ 1396.713214] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:27:07 (1788272827) [ 1398.466862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg348-server [ 1400.276101] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:27:10 (1788272830) [ 1402.349683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg348-server [ 1404.167747] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:27:14 (1788272834) [ 1405.801912] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg348-server [ 1407.268714] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:27:18 (1788272838) [ 1408.794956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg348-server [ 1410.424133] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:27:21 (1788272841) [ 1412.283844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg348-server [ 1414.152885] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:27:24 (1788272844) [ 1416.154239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg348-server [ 1417.667116] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:27:28 (1788272848) [ 1417.936817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1417.959962] LNet: 83020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1417.971659] LNetError: 83020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.979966] LNet: Removed LNI 192.168.203.48@tcp [ 1419.054664] LNet: Removed LNI 192.168.203.48@tcp1 [ 1419.105405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.159266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.183412] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1419.824321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1419.893868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1420.048200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.076171] LNet: 83167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.081333] LNetError: 83167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.084992] LNetError: 83167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1420.100837] LNet: Removed LNI 192.168.203.48@tcp [ 1420.112929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.153823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1420.172289] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1420.842698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1421.055760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.127605] LNet: Removed LNI 192.168.203.48@tcp [ 1422.174499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.234815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.258648] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1422.859991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1422.947390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1423.167196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1423.195634] LNet: 83459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.205965] LNet: 83459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1423.215487] LNetError: 83459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.225342] LNetError: 83459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1423.239969] LNet: Removed LNI 192.168.203.48@tcp [ 1423.266790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.341700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1423.373994] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1424.247164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1424.510697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1425.574292] LNet: Removed LNI 192.168.203.48@tcp [ 1425.603867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1425.699781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1425.735749] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1426.815616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1426.913527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1427.208122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1427.230485] LNet: 83752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.240540] LNet: 83752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1427.250819] LNetError: 83752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.256678] LNetError: 83752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1428.334471] LNet: Removed LNI 192.168.203.48@tcp [ 1428.365831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.432931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1428.474926] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1429.264367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1430.779519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.846228] LNet: Removed LNI 192.168.203.48@tcp [ 1431.902994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.991090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1432.036624] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1433.074372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1433.195699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1433.544524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1433.584165] LNetError: 84054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.589269] LNetError: 84054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1434.662549] LNet: Removed LNI 192.168.203.48@tcp [ 1434.721706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.804840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1434.844293] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1435.942659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1437.451701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1437.504641] LNet: 84209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.511921] LNet: 84209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1438.574623] LNet: Removed LNI 192.168.203.48@tcp [ 1438.618632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.741365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1438.781179] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1439.892745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1439.982102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1440.290513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1441.390144] LNet: Removed LNI 192.168.203.48@tcp [ 1441.430470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.513181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1441.544775] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1442.366190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1442.744452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1442.778248] LNetError: 84502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.792613] LNetError: 84502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1443.814044] LNet: Removed LNI 192.168.203.48@tcp [ 1443.881405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1443.958700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1443.985379] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1444.719332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1444.842237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1445.219631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1446.309750] LNet: Removed LNI 192.168.203.48@tcp [ 1446.361312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.464724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1446.489159] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1447.653938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1449.326210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.408188] LNet: Removed LNI 192.168.203.48@tcp [ 1450.476219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.556750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.581080] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1451.352826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1451.457959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.495566] LNet: Removed LNI 192.168.203.48@tcp [ 1451.520705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.576959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1451.609883] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1452.509425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1452.603917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.644782] LNet: Removed LNI 192.168.203.48@tcp [ 1452.666388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.768084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1452.941892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.992427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.069857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1453.193618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1453.236614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.320821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1453.444946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1453.478095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.530612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1453.573468] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1454.499572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.524831] LNet: 85244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.534315] LNet: 85244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1454.552408] LNet: Removed LNI 192.168.203.48@tcp [ 1454.587342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.684661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1461.682888] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:28:12 (1788272892) [ 1461.766436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.825645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.940575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1461.973067] LNet: Added LNI 192.168.203.48@tcp231 [8/256/0/180] [ 1463.311639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.357326] LNetError: 85964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.370367] LNetError: 85964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1463.393866] LNet: Removed LNI 192.168.203.48@tcp231 [ 1463.426515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.515748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1463.555968] LNet: Added LNI 192.168.203.48@tcp231 [8/256/0/180] [ 1463.559950] LNet: 85972:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1464.112356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1464.154819] LNet: Removed LNI 192.168.203.48@tcp231 [ 1464.182115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1464.222508] LNet: Added LNI 192.168.203.48@tcp231 [8/256/0/180] [ 1465.520513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1465.572863] LNet: Removed LNI 192.168.203.48@tcp231 [ 1465.596732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.701873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1465.721969] LNet: Added LNI 192.168.203.48@tcp231 [8/256/0/0] [ 1465.726983] LNet: 86125:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1466.247987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1466.301335] LNet: Removed LNI 192.168.203.48@tcp231 [ 1466.327625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1466.373935] LNet: Added LNI 192.168.203.48@tcp231 [8/256/0/0] [ 1467.567871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1467.592865] LNet: Removed LNI 192.168.203.48@tcp231 [ 1467.632133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.714742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1467.746966] LNet: Added LNI 192.168.203.48@tcp231 [8/256/0/60] [ 1467.754867] LNet: 86276:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1468.254242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1468.310845] LNet: Removed LNI 192.168.203.48@tcp231 [ 1468.332623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1468.384108] LNet: Added LNI 192.168.203.48@tcp231 [8/256/0/60] [ 1469.478939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.505092] LNet: Removed LNI 192.168.203.48@tcp231 [ 1469.522364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.602879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1469.630962] LNet: Added LNI 192.168.203.48@tcp231 [8/256/0/180] [ 1470.114713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1470.146959] LNet: Removed LNI 192.168.203.48@tcp231 [ 1470.161446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1470.190130] LNet: Added LNI 192.168.203.48@tcp231 [8/256/0/180] [ 1478.021765] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:28:28 (1788272908) [ 1478.095447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.147207] LNet: Removed LNI 192.168.203.48@tcp231 [ 1478.165962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.238070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1478.266270] LNet: Added LNI 192.168.203.48@tcp232 [8/256/0/180] [ 1479.034977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1479.095790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1479.144395] LNet: Removed LNI 192.168.203.48@tcp232 [ 1479.168457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1479.207364] LNet: Added LNI 192.168.203.48@tcp232 [8/256/0/180] [ 1480.013073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1480.048769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1480.086148] LNet: Removed LNI 192.168.203.48@tcp232 [ 1480.126106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1480.155749] LNet: Added LNI 192.168.203.48@tcp232 [8/256/0/180] [ 1480.814667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1480.852420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1480.882780] LNet: Removed LNI 192.168.203.48@tcp232 [ 1480.899653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1480.928789] LNet: Added LNI 192.168.203.48@tcp232 [8/256/0/180] [ 1481.681230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1481.722840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1481.757923] LNet: Removed LNI 192.168.203.48@tcp232 [ 1481.769744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1487.564505] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:28:38 (1788272918) [ 1487.991902] LNet: 88266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.008788] LNet: 88266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1488.693069] Key type .llcrypt unregistered [ 1488.701165] Key type ._llcrypt unregistered [ 1494.845652] Key type ._llcrypt registered [ 1494.848926] Key type .llcrypt registered [ 1495.093431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1495.114363] alg: No test for adler32 (adler32-zlib) [ 1496.047227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1496.077690] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1502.816238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 1511.217439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.277135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1511.311292] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1512.535195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_interrupt -r 2 [ 1512.573700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_interrupt -r 2 [ 1512.618277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_interrupt -r 2 [ 1512.653421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_interrupt -r 2 [ 1512.701407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_interrupt -r 2 [ 1512.741856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_interrupt -r 2 [ 1512.784345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_interrupt -r 2 [ 1512.828180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_interrupt -r 2 [ 1512.865988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.48@tcp 192.168.203.148@tcp [ 1512.893872] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1512.902504] LNet: 1 local NIs in recovery (showing 1): 192.168.203.48@tcp [ 1513.101863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1513.129805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1513.163472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1513.190615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1513.217604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1513.244742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1513.282649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1513.311283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1513.327827] LNet: 88933:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1513.345470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1513.373055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1513.416314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1513.460765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1513.486669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1513.517265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1513.982500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_dropped -r 2 [ 1514.009433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_dropped -r 2 [ 1514.035147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_dropped -r 2 [ 1514.067704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_dropped -r 2 [ 1514.115162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_dropped -r 2 [ 1514.151658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_dropped -r 2 [ 1514.181753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_dropped -r 2 [ 1514.210731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_dropped -r 2 [ 1514.242534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.48@tcp 192.168.203.148@tcp [ 1514.267452] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1514.525852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1514.558302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1514.606681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1514.650983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1514.701032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1514.731431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1514.764528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1514.808724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1514.838185] LNet: 89053:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1514.871594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1514.908429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1514.939154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1514.974566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1515.004475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1515.037651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1515.515113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_aborted -r 2 [ 1515.565837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_aborted -r 2 [ 1515.619247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_aborted -r 2 [ 1515.669954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_aborted -r 2 [ 1515.721460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_aborted -r 2 [ 1515.784565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_aborted -r 2 [ 1515.830182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_aborted -r 2 [ 1515.871945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_aborted -r 2 [ 1515.907174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.48@tcp 192.168.203.148@tcp [ 1515.925217] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1516.208575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1516.258760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1516.294770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1516.324409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1516.355983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1516.393991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1516.445222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1516.489278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1516.509692] LNet: 89174:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1516.534843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1516.562979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1516.589746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1516.619304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1516.648440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1516.678307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1517.172978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_no_route -r 2 [ 1517.204858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_no_route -r 2 [ 1517.249142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_no_route -r 2 [ 1517.285443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_no_route -r 2 [ 1517.331145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_no_route -r 2 [ 1517.381570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_no_route -r 2 [ 1517.434709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_no_route -r 2 [ 1517.473033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_no_route -r 2 [ 1517.505347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.48@tcp 192.168.203.148@tcp [ 1517.774495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1517.802663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1517.836429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1517.865265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1517.916563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1517.978536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1518.008976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1518.062540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1518.098612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1518.129416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1518.179086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1518.203453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1518.243591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1518.271274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1518.697275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e local_timeout -r 2 [ 1518.733960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e local_timeout -r 2 [ 1518.764262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e local_timeout -r 2 [ 1518.794500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e local_timeout -r 2 [ 1518.828543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e local_timeout -r 2 [ 1518.856882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e local_timeout -r 2 [ 1518.890739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e local_timeout -r 2 [ 1518.937503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e local_timeout -r 2 [ 1518.986475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.48@tcp 192.168.203.148@tcp [ 1519.010016] LNet: There was an unexpected network error while writing to 192.168.203.148: rc = -22 [ 1519.014711] LNet: Skipped 1 previous similar message [ 1519.279634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1519.314957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1519.363992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1519.401606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1519.432993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1519.478513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1519.553208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1519.595584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1519.621906] LNet: 89414:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1519.632210] LNet: 89414:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1519.658933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1519.686055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1519.727766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.755177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1519.796578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1519.853280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1520.440450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp -m GET -e remote_dropped -r 2 [ 1520.478175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp -d 192.168.203.148@tcp1 -m GET -e remote_dropped -r 2 [ 1520.509470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp -m GET -e remote_dropped -r 2 [ 1520.562461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp -d 192.168.203.48@tcp1 -m GET -e remote_dropped -r 2 [ 1520.594493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp -m GET -e remote_dropped -r 2 [ 1520.621347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.148@tcp1 -m GET -e remote_dropped -r 2 [ 1520.658978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp -m GET -e remote_dropped -r 2 [ 1520.693598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.48@tcp1 -d 192.168.203.48@tcp1 -m GET -e remote_dropped -r 2 [ 1520.729539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.48@tcp 192.168.203.148@tcp [ 1520.754356] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.148@tcp [ 1520.932779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1520.961839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1521.002607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1521.037583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1521.094747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1521.129792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1521.177348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1521.227119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1521.272657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1521.312862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1521.367136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1521.406504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1521.448971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1521.492979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1526.238172] LNet: 89612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.249223] LNetError: 89612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.260324] LNet: Removed LNI 192.168.203.48@tcp [ 1526.295603] LNet: Removed LNI 192.168.203.48@tcp1 [ 1526.786863] Key type .llcrypt unregistered [ 1526.793822] Key type ._llcrypt unregistered [ 1542.259365] Key type ._llcrypt registered [ 1542.262383] Key type .llcrypt registered [ 1543.807105] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:29:34 (1788272974) [ 1544.287724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1544.315517] alg: No test for adler32 (adler32-zlib) [ 1545.150827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.209293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.244613] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1546.577892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1546.640587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.48@tcp [ 1546.679508] LNetError: 90591:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.48@tcp to recovery [ 1546.810886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.48@tcp [ 1546.946646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1554.378956] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:29:44 (1788272984) [ 1554.533624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.564707] LNet: 91177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.571906] LNetError: 91177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.607234] LNet: Removed LNI 192.168.203.48@tcp [ 1554.652456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.715889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.741501] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1555.509341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1555.543652] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1556.781684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.48@tcp [ 1556.800657] LNetError: 91448:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.48@tcp to recovery [ 1557.791378] LNet: 1 local NIs in recovery (showing 1): 192.168.203.48@tcp [ 1557.794415] LNet: There was an unexpected network error while writing to 192.168.203.48: rc = -22 [ 1558.092391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.48@tcp [ 1566.115704] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:29:57 (1788272997) [ 1568.275685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg348-server [ 1569.573585] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:30:00 (1788273000) [ 1571.168859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg348-server [ 1573.036700] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:30:03 (1788273003) [ 1575.078375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg348-server [ 1577.131744] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:30:07 (1788273007) [ 1577.269292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.301037] LNet: 93073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.308880] LNetError: 93073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.311833] LNet: Removed LNI 192.168.203.48@tcp [ 1577.320941] LNet: Removed LNI 192.168.203.48@tcp1 [ 1577.342792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.390570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1577.409327] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1578.203069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1578.279195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1578.326063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1578.453732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1578.503546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1578.582682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1586.920864] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:30:17 (1788273017) [ 1587.149736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.221227] LNet: 93811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.232709] LNetError: 93811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.248824] LNetError: 93811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1587.272807] LNet: Removed LNI 192.168.203.48@tcp [ 1587.356743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.439622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1587.496874] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1596.423511] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:30:27 (1788273027) [ 1596.488864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.517954] LNet: 94509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.524839] LNetError: 94509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.557944] LNet: Removed LNI 192.168.203.48@tcp [ 1596.592469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.690731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.714876] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1605.118954] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:30:35 (1788273035) [ 1606.776135] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1608.577570] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:30:39 (1788273039) [ 1609.189390] LNet: 95550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.202840] LNetError: 95550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.235308] LNet: Removed LNI 192.168.203.48@tcp [ 1610.145261] Key type .llcrypt unregistered [ 1610.149120] Key type ._llcrypt unregistered [ 1616.155956] Key type ._llcrypt registered [ 1616.158581] Key type .llcrypt registered [ 1616.330761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1616.354875] alg: No test for adler32 (adler32-zlib) [ 1617.341449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1617.393184] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1624.201914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 1628.538047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.148@tcp [ 1628.955335] LNet: 95976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.962830] LNetError: 95976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.982375] LNet: Removed LNI 192.168.203.48@tcp [ 1629.358396] Key type .llcrypt unregistered [ 1629.360277] Key type ._llcrypt unregistered [ 1640.370926] Key type ._llcrypt registered [ 1640.374522] Key type .llcrypt registered [ 1641.688087] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1643.341097] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1645.188923] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:31:15 (1788273075) [ 1646.757689] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1648.980879] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:31:19 (1788273079) [ 1651.082683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg348-server [ 1653.291518] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:31:23 (1788273083) [ 1655.609182] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg348-server [ 1657.591293] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:31:28 (1788273088) [ 1659.133885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1659.179954] alg: No test for adler32 (adler32-zlib) [ 1660.266254] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 1660.569706] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1662.263376] Key type lgssc registered [ 1663.615515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1684.736363] Key type lgssc unregistered [ 1685.037325] LNet: 99155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.045123] LNetError: 99155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.074305] LNet: Removed LNI 192.168.203.48@tcp [ 1685.910533] Key type .llcrypt unregistered [ 1685.914797] Key type ._llcrypt unregistered [ 1699.645705] Key type ._llcrypt registered [ 1699.648653] Key type .llcrypt registered [ 1701.281563] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:32:11 (1788273131) [ 1701.713057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1701.751588] alg: No test for adler32 (adler32-zlib) [ 1702.749587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.840826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1702.905739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1702.974334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1703.051318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1710.833557] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:32:20 (1788273140) [ 1711.149941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.190481] LNet: 100572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1711.230197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.325816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1711.387322] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1712.927826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.950883] LNet: 100717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.957545] LNetError: 100717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.984264] LNet: Removed LNI 192.168.203.48@tcp [ 1713.016672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.097731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1713.158821] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1714.382992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.408621] LNet: 100745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.421495] LNetError: 100745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.461759] LNet: Removed LNI 192.168.203.48@tcp [ 1714.497285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.568814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1714.610364] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1721.879547] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:32:32 (1788273152) [ 1722.060882] LNet: 101329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.069169] LNetError: 101329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.105315] LNet: Removed LNI 192.168.203.48@tcp [ 1723.154851] Key type .llcrypt unregistered [ 1723.163738] Key type ._llcrypt unregistered [ 1730.475710] Key type ._llcrypt registered [ 1730.479721] Key type .llcrypt registered [ 1730.692647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1730.717868] alg: No test for adler32 (adler32-zlib) [ 1731.620272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.684506] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1731.687993] LNet: *** cfs_fail_loc=e003, val=100*** [ 1732.437642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1732.461942] LNet: *** cfs_fail_loc=e003, val=100*** [ 1732.466136] LNet: Skipped 2 previous similar messages [ 1732.476396] Ping reply received with mapping: [ 1732.481375] found metadata reply NI[192.168.203.48@tcp] [ 1732.484989] 30 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1732.490524] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1732.494174] 5a 5a 5a 5a 5a 5a 5a 5a [ 1732.650431] LNet: 101737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.659434] LNetError: 101737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.674854] LNet: Removed LNI 192.168.203.48@tcp [ 1733.230502] Key type .llcrypt unregistered [ 1733.232556] Key type ._llcrypt unregistered [ 1745.684192] Key type ._llcrypt registered [ 1745.685687] Key type .llcrypt registered [ 1746.894498] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:32:57 (1788273177) [ 1747.605392] Key type .llcrypt unregistered [ 1747.607771] Key type ._llcrypt unregistered [ 1753.582736] Key type ._llcrypt registered [ 1753.584479] Key type .llcrypt registered [ 1753.733847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1753.749389] alg: No test for adler32 (adler32-zlib) [ 1754.917924] LNetError: 102811:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1755.539291] Key type .llcrypt unregistered [ 1755.543102] Key type ._llcrypt unregistered [ 1761.427943] Key type ._llcrypt registered [ 1761.429673] Key type .llcrypt registered [ 1762.822169] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:33:13 (1788273193) [ 1763.722498] Key type .llcrypt unregistered [ 1763.731249] Key type ._llcrypt unregistered [ 1770.449657] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1770.453863] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1773.234299] Key type ._llcrypt registered [ 1773.239797] Key type .llcrypt registered [ 1773.506497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1773.520666] alg: No test for adler32 (adler32-zlib) [ 1774.495351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.544188] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1783.527876] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:33:34 (1788273214) [ 1783.797151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.841201] LNet: 104599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.849660] LNetError: 104599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1783.864607] LNet: Removed LNI 10.1.2.3@tcp [ 1783.890619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.691545] Key type .llcrypt unregistered [ 1784.694537] Key type ._llcrypt unregistered [ 1790.898218] Key type ._llcrypt registered [ 1790.902507] Key type .llcrypt registered [ 1791.162728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1791.194045] alg: No test for adler32 (adler32-zlib) [ 1792.122196] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1792.184480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1792.336510] LNet: 104891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.342936] LNetError: 104891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1792.372333] LNet: Removed LNI 192.168.203.48@tcp [ 1793.140828] Key type .llcrypt unregistered [ 1793.142468] Key type ._llcrypt unregistered [ 1803.599709] Key type ._llcrypt registered [ 1803.603023] Key type .llcrypt registered [ 1804.853688] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:33:55 (1788273235) [ 1805.632324] Key type .llcrypt unregistered [ 1805.637405] Key type ._llcrypt unregistered [ 1811.467666] Key type ._llcrypt registered [ 1811.470482] Key type .llcrypt registered [ 1811.625613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1811.639886] alg: No test for adler32 (adler32-zlib) [ 1818.654581] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:34:09 (1788273249) [ 1818.826418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.870509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.932884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1818.955520] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1819.636873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1819.681370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1819.714867] LNet: Removed LNI 192.168.203.48@tcp [ 1819.955502] LNet: 106646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.311490] Key type .llcrypt unregistered [ 1820.314564] Key type ._llcrypt unregistered [ 1831.827319] Key type ._llcrypt registered [ 1831.831516] Key type .llcrypt registered [ 1833.357942] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:34:23 (1788273263) [ 1833.881791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.902655] alg: No test for adler32 (adler32-zlib) [ 1834.893306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.964989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.991383] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1835.866190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1835.931790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.48@tcp [ 1835.958143] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1835.960860] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1835.972965] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1835, 0, 0 [ 1835.982259] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1835.996290] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1836.004747] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1836.009800] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1835, 0, 0 [ 1836.014442] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1836.025664] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1836.031530] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1836.034439] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1835, 0, 1885 [ 1836.038857] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1836.041240] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1836.045241] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1836.049235] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1835, 0, 1885 [ 1836.053909] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1842.504969] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:34:33 (1788273273) [ 1843.038967] LNet: 108184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.046848] LNetError: 108184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.066866] LNet: Removed LNI 192.168.203.48@tcp [ 1843.612422] Key type .llcrypt unregistered [ 1843.617095] Key type ._llcrypt unregistered [ 1849.355460] Key type ._llcrypt registered [ 1849.357688] Key type .llcrypt registered [ 1849.568375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1849.585339] alg: No test for adler32 (adler32-zlib) [ 1850.502445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1850.526923] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1857.686913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 1865.913091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.974797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1865.997340] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 1867.057462] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1867.196190] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1867.203809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1867.210737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1869.632139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1869.658160] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1880.863162] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:35:11 (1788273311) [ 1880.953680] LNet: 109688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.962446] LNetError: 109688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.973986] LNet: Removed LNI 192.168.203.48@tcp [ 1880.989708] LNet: Removed LNI 192.168.203.48@tcp1 [ 1881.012729] LNet: Removed LNI 10.1.2.3@tcp99 [ 1881.794816] Key type .llcrypt unregistered [ 1881.798930] Key type ._llcrypt unregistered [ 1888.744896] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1888.752846] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1891.510040] Key type ._llcrypt registered [ 1891.512018] Key type .llcrypt registered [ 1891.752501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1891.786659] alg: No test for adler32 (adler32-zlib) [ 1892.685036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1892.735645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1892.759576] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1893.405905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1893.423919] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1895.350193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1895.389153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1895.432733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1895.457749] LNetError: 110446:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1895.488441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1895.530626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1895.944346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1896.013355] LNet: 110466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.018951] LNetError: 110466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1896.024145] LNet: Removed LNI 10.1.2.3@tcp [ 1897.061728] LNet: Removed LNI 192.168.203.48@tcp [ 1897.701619] Key type .llcrypt unregistered [ 1897.705638] Key type ._llcrypt unregistered [ 1908.977457] Key type ._llcrypt registered [ 1908.979330] Key type .llcrypt registered [ 1910.289165] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:35:40 (1788273340) [ 1910.636735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1910.649352] alg: No test for adler32 (adler32-zlib) [ 1911.530956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.591446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1911.622728] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1912.267175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 1912.349296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg348-client@tcp [ 1918.289167] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:35:49 (1788273349) [ 1918.373646] LNet: 111994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.385184] LNetError: 111994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.403956] LNet: Removed LNI 192.168.203.48@tcp [ 1920.232594] Key type .llcrypt unregistered [ 1920.238039] Key type ._llcrypt unregistered [ 1925.923559] Key type ._llcrypt registered [ 1925.931962] Key type .llcrypt registered [ 1926.220751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1926.267279] alg: No test for adler32 (adler32-zlib) [ 1927.176661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1927.214067] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1927.241736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1932.982477] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:36:03 (1788273363) [ 1934.114794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1935.618569] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:36:06 (1788273366) [ 1937.218730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1939.056590] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:36:09 (1788273369) [ 1939.203654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1939.261953] LNet: 113422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.274395] LNetError: 113422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.297452] LNet: Removed LNI 192.168.203.48@tcp [ 1939.328792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.411288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1939.435279] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1940.189978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1940.230993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1946.106591] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:36:16 (1788273376) [ 1946.286966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.330342] LNet: 114137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.337272] LNetError: 114137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.353919] LNet: Removed LNI 192.168.203.48@tcp [ 1946.368969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.420456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1946.454687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1947.167369] Key type .llcrypt unregistered [ 1947.169677] Key type ._llcrypt unregistered [ 1958.703521] Key type ._llcrypt registered [ 1958.705937] Key type .llcrypt registered [ 1960.058270] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:36:30 (1788273390) [ 1960.543582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1960.555469] alg: No test for adler32 (adler32-zlib) [ 1961.533262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1961.615600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1961.648893] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1962.291790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1962.355847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1968.299658] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:36:39 (1788273399) [ 1968.472376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.510305] LNet: 115695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.517795] LNetError: 115695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.556759] LNet: Removed LNI 192.168.203.48@tcp [ 1968.599994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1968.675347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1968.728621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1975.076157] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:36:45 (1788273405) [ 1975.189818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.213116] LNet: 116274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.232880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.309919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1975.633137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1975.659452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1975.735183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1976.233975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1976.273853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1976.362229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.377525] LNet: 116310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.396715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.505547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1977.123383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1977.185820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1977.279245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1977.762930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1977.818815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1983.591605] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:36:54 (1788273414) [ 1983.819292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1983.849797] LNet: 116910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.873836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1983.922558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1983.943708] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1984.570315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1984.597545] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 1985.585874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1985.661941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1985.879086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.898957] LNetError: 117189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.905346] LNet: Removed LNI 192.168.203.48@tcp [ 1985.918633] LNet: Removed LNI 192.168.203.48@tcp2 [ 1985.938243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.998093] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1986.006711] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 1986.011454] LNet: 117195:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1986.449269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1986.587873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1987.615346] LNet: Removed LNI 192.168.203.48@tcp [ 1987.618235] LNetError: 117232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1987.624255] LNetError: 117232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1987.641944] LNet: Removed LNI 192.168.203.48@tcp2 [ 1987.702140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.805093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1987.839606] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1988.392320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1988.409906] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 1989.060660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1989.104446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1989.176981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1989.371642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.399293] LNet: 117517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.405797] LNet: 117517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1990.431371] LNet: Removed LNI 192.168.203.48@tcp [ 1990.435778] LNetError: 117517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.441760] LNetError: 117517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1990.457268] LNet: Removed LNI 192.168.203.48@tcp2 [ 1990.482937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.591466] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1990.598135] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 1990.611530] LNet: 117523:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1991.278439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1991.572547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1991.608733] LNet: Removed LNI 192.168.203.48@tcp [ 1992.680086] LNet: Removed LNI 192.168.203.48@tcp2 [ 1992.705510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.763141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1992.791838] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1993.424556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1993.444898] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 1994.149204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1994.193183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1994.410858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.487663] LNet: Removed LNI 192.168.203.48@tcp [ 1995.494178] LNetError: 117840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1995.502873] LNetError: 117840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1995.521916] LNet: Removed LNI 192.168.203.48@tcp2 [ 1995.560787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1995.614495] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1995.622763] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 1995.626454] LNet: 117846:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1995.971433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1996.119526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1997.154296] LNet: Removed LNI 192.168.203.48@tcp [ 1997.173812] LNet: Removed LNI 192.168.203.48@tcp2 [ 1997.211739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.276788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1997.315149] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 1997.900888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1997.923689] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 1998.645880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1998.703364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1998.768979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 1998.936722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1998.957473] LNet: 118167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.962660] LNet: 118167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2000.031188] LNet: Removed LNI 192.168.203.48@tcp [ 2001.063323] LNet: Removed LNI 192.168.203.48@tcp2 [ 2001.104351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.206788] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2001.216428] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 2001.230729] LNet: 118173:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2001.855907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 2002.017494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.035920] LNet: Removed LNI 192.168.203.48@tcp [ 2003.048724] LNet: Removed LNI 192.168.203.48@tcp2 [ 2003.072647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.178955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2003.225898] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2004.286790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2004.318246] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 2005.072593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2005.100472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2005.172562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 2005.567383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.597290] LNetError: 118507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.602477] LNetError: 118507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2005.609937] LNet: Removed LNI 192.168.203.48@tcp [ 2005.623227] LNet: Removed LNI 192.168.203.48@tcp2 [ 2005.645508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.717561] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2005.727944] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 2005.741609] LNet: 118513:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2006.290176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 2006.681442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.776292] LNet: Removed LNI 192.168.203.48@tcp [ 2007.799476] LNet: Removed LNI 192.168.203.48@tcp2 [ 2007.836850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.881358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2007.902772] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2008.663771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2008.703656] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 2009.580990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 2009.638836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2009.678554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2009.767520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 2010.186246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2011.231472] LNet: Removed LNI 192.168.203.48@tcp [ 2012.262833] LNet: Removed LNI 192.168.203.48@tcp2 [ 2012.301335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.366780] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2012.377353] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 2012.849974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 2019.500883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2021.080883] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:37:31 (1788273451) [ 2021.294677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2021.334297] LNet: 119634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.343442] LNet: 119634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2021.348448] LNet: Removed LNI 192.168.203.48@tcp [ 2022.388967] LNet: Removed LNI 192.168.203.48@tcp2 [ 2022.423360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.809929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2022.857621] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2023.620587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2023.645707] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2024.540031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2024.567625] LNetError: 119908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2024.575716] LNetError: 119908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2024.586211] LNet: Removed LNI 10.1.2.3@tcp [ 2024.602091] LNet: Removed LNI 192.168.203.48@tcp [ 2024.624190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.700527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2024.730946] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2024.738991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2025.248850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2025.282355] LNet: Removed LNI 192.168.203.48@tcp [ 2025.306149] LNet: Removed LNI 10.1.2.3@tcp [ 2032.084422] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:37:42 (1788273462) [ 2032.369529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.435753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.512897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2032.539964] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2033.242647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2033.293336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 2033.341317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 2033.371892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 2033.401135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 2033.436468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 2033.475352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 2033.510340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 2033.541927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 2033.577107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 2033.637313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.48@tcp [ 2033.817160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2039.409125] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:37:50 (1788273470) [ 2039.580801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2040.677802] LNet: Removed LNI 192.168.203.48@tcp [ 2040.694833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.819218] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2040.933851] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2040.942694] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2040.951504] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2043.331466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2043.352852] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2044.075458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2044.113358] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2046.529440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 2046.581893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2046.674839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.48@tcp -m PUT -e local_timeout -r 1 [ 2046.763936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2046.872822] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2046.889102] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2047.210551] LNetError: 121313:0:(lib-move.c:4801:lnet_parse()) 192.168.203.48@tcp, src 192.168.203.48@tcp: Dropping PUT (error -108 looking up sender) [ 2048.224484] LNet: Removed LNI 10.1.2.3@tcp [ 2049.263574] LNet: Removed LNI 192.168.203.48@tcp [ 2049.776769] Key type .llcrypt unregistered [ 2049.780633] Key type ._llcrypt unregistered [ 2061.297917] Key type ._llcrypt registered [ 2061.300401] Key type .llcrypt registered [ 2062.457399] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:38:13 (1788273493) [ 2062.791476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.806290] alg: No test for adler32 (adler32-zlib) [ 2063.734714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.047136] LNet: 122628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.639056] Key type .llcrypt unregistered [ 2064.643121] Key type ._llcrypt unregistered [ 2071.298812] Key type ._llcrypt registered [ 2071.301352] Key type .llcrypt registered [ 2071.439988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2071.451526] alg: No test for adler32 (adler32-zlib) [ 2072.367944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2072.413306] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2078.730785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 2078.930806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2079.466182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.148@tcp [ 2079.588280] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.148@tcp [ 2079.720782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2079.761771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2079.799539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2079.835976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2079.877298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2079.911379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2079.940757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2079.975280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2079.992484] LNet: 123081:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2080.009788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2080.036196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2080.075921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2080.103498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2080.146717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2080.181342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2080.716775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2080.926494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2080.949718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2080.982644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2081.020589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2081.058516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2081.096381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2081.147316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2081.208788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2081.237178] LNet: 123198:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2081.259949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2081.315120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2081.355678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2081.398784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2081.443713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2081.481555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2082.047595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.148@tcp [ 2082.328208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2082.379121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2082.414543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2082.456630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2082.496622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2082.529802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2082.557772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2082.598467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2082.637423] LNet: 123315:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2082.665349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2082.699411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2082.724657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2082.745355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2082.774099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2082.797757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2083.314936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2083.548080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2083.570947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2083.595473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2083.621949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2083.646283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2083.670794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2083.697426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2083.730490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2083.783242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2083.818731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2083.855509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2083.881518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2083.910272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2083.933550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2084.199688] LNet: 123483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2084.205532] LNetError: 123483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.221050] LNet: Removed LNI 192.168.203.48@tcp [ 2084.600680] Key type .llcrypt unregistered [ 2084.604451] Key type ._llcrypt unregistered [ 2097.889436] Key type ._llcrypt registered [ 2097.891630] Key type .llcrypt registered [ 2098.970553] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:38:49 (1788273529) [ 2099.373223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.386444] alg: No test for adler32 (adler32-zlib) [ 2100.329341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.733625] LNet: 124370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.414520] Key type .llcrypt unregistered [ 2101.416845] Key type ._llcrypt unregistered [ 2106.409401] Key type ._llcrypt registered [ 2106.414861] Key type .llcrypt registered [ 2106.681137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2106.701043] alg: No test for adler32 (adler32-zlib) [ 2107.655625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2107.699423] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2114.116138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.148@tcp [ 2122.203438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.277249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2122.324532] LNet: Added LNI 192.168.203.48@tcp1 [8/256/0/180] [ 2123.076528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2123.545354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.148@tcp [ 2123.791341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2123.829243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2123.861454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2123.886438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2123.914622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2123.938438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2123.968123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2123.997239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2124.018723] LNet: 125025:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2124.036838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2124.063221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2124.110573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2124.129810] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.148@tcp [ 2124.154314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2124.197701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2124.221912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2124.745502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2124.947913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2124.971946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2124.994656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2125.019456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2125.044938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2125.070577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2125.094451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2125.113350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2125.126861] LNet: 125142:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2125.143420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2125.167103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2125.198080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2125.233861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2125.271534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2125.296372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2125.782334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.148@tcp [ 2125.985597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2126.010225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2126.033913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2126.057865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2126.081761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2126.106450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2126.135749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2126.157168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2126.173635] LNet: 125260:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2126.189435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2126.215243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2126.244476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2126.272299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2126.296921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2126.324495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2126.859270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2127.073518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2127.111901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2127.140051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2127.166553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2127.196344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2127.221822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2127.248328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2127.279494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2127.307930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2127.338154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2127.384431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2127.421659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2127.458893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2127.498314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2131.872681] LNet: 125461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.879833] LNetError: 125461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.885328] LNet: Removed LNI 192.168.203.48@tcp [ 2131.897369] LNet: Removed LNI 192.168.203.48@tcp1 [ 2132.270929] Key type .llcrypt unregistered [ 2132.272657] Key type ._llcrypt unregistered [ 2147.004857] Key type ._llcrypt registered [ 2147.006345] Key type .llcrypt registered [ 2148.562842] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:39:38 (1788273578) [ 2149.276041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2149.292546] alg: No test for adler32 (adler32-zlib) [ 2150.184954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.254842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.281880] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2151.155299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2151.172942] LNet: Added LNI 192.168.203.48@tcp2 [8/256/0/180] [ 2152.207773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.48@tcp [ 2152.291351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2153.378123] LNet: Removed LNI 192.168.203.48@tcp [ 2160.413452] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:39:51 (1788273591) [ 2161.993814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2164.791719] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:39:54 (1788273594) [ 2172.712350] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:40:03 (1788273603) [ 2178.741472] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:40:09 (1788273609) [ 2184.955397] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:40:15 (1788273615) [ 2191.519286] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:40:22 (1788273622) [ 2196.982540] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:40:27 (1788273627) [ 2203.876045] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:40:34 (1788273634) [ 2209.517147] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:40:40 (1788273640) [ 2215.757087] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:40:46 (1788273646) [ 2221.181744] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:40:52 (1788273652) [ 2226.501788] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:40:57 (1788273657) [ 2232.696059] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:41:03 (1788273663) [ 2237.845409] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:41:08 (1788273668) [ 2243.060383] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:41:14 (1788273674) [ 2249.014363] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:41:19 (1788273679) [ 2254.549541] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:41:25 (1788273685) [ 2259.807476] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:41:30 (1788273690) [ 2266.536865] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:41:37 (1788273697) [ 2266.779694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2266.805452] LNet: 137151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2266.816507] LNetError: 137151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.838162] LNet: Removed LNI 192.168.203.48@tcp2 [ 2266.874519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.061246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.114537] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2268.219990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.245919] LNet: 137293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.253635] LNetError: 137293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.274450] LNet: Removed LNI 192.168.203.48@tcp [ 2268.306499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.378233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2268.402882] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2273.763282] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:41:44 (1788273704) [ 2279.042918] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:41:50 (1788273710) [ 2283.776146] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:41:54 (1788273714) [ 2289.365263] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:42:00 (1788273720) [ 2295.051623] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:42:05 (1788273725) [ 2301.022886] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:42:11 (1788273731) [ 2306.695857] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:42:17 (1788273737) [ 2312.557591] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:42:23 (1788273743) [ 2317.512287] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:42:28 (1788273748) [ 2322.584376] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:42:33 (1788273753) [ 2327.347553] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:42:38 (1788273758) [ 2332.420629] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:42:43 (1788273763) [ 2337.445628] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2132 sec ========== 10:42:48 (1788273768) [ 2337.652655] LNet: 144755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.657667] LNetError: 144755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.671666] LNet: Removed LNI 192.168.203.48@tcp [ 2338.152836] Key type .llcrypt unregistered [ 2338.155808] Key type ._llcrypt unregistered [ 2348.012176] Key type ._llcrypt registered [ 2348.016692] Key type .llcrypt registered [ 2348.216127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2348.230361] alg: No test for adler32 (adler32-zlib) [ 2349.250269] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 2349.460402] LNet: Added LNI 192.168.203.48@tcp [8/256/0/180] [ 2351.112230] Key type lgssc registered [ 2352.232488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2397.819480] Lustre: Mounted lustre-client - version 2.17.57_64_g65721f0 [ 2402.127697] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2408.844161] Lustre: Unmounted lustre-client [ 2444.213214] Key type lgssc unregistered [ 2444.420198] LNet: 147712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.426709] LNetError: 147712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.447611] LNet: Removed LNI 192.168.203.48@tcp [ 2445.086577] Key type .llcrypt unregistered [ 2445.092191] Key type ._llcrypt unregistered